[2023-09-07 16:30:00,603 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 16:30:00,754 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:30:00,754 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:30:00,755 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:30:00,755 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:30:00,755 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:30:00,755 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 16:30:00,777 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:30:00,789 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 16:30:00,790 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 16:30:00,790 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 16:30:00,790 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 16:30:00,977 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 16:30:00,977 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 16:30:00,979 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 16:30:00,985 ERROR][11][wan-tasty-kitalpha][T_1] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 16:30:00,986 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:30:01,001 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:30:01,001 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:30:01,001 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:30:01,002 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:30:01,002 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:30:01,002 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 16:30:01,023 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:30:01,106 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:01,269 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:01,416 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:01,527 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:30:01,529 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2449 [2023-09-07 16:30:01,529 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:30:01,530 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:30:01,530 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:30:01,530 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:30:01,531 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:30:01,537 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:30:01,539 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:30:01.539844 [2023-09-07 16:30:01,547 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:01,658 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:30:01,659 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2457 [2023-09-07 16:30:01,660 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:30:01,660 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:30:01,661 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:30:01,661 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:30:01,662 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:30:01,668 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:30:01,670 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:30:01.670202 [2023-09-07 16:30:01,708 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:01,733 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:30:01,765 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:30:01,767 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:30:01,775 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:30:01,775 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694104201.7320113, 'te': 1694104201.7387872, 'initial_status': True} ====================================================================================================  [2023-09-07 16:30:01,777 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:30:01.777314 [2023-09-07 16:30:01,782 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:30:01,782 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:30:01,939 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:01,974 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:30:02,014 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:30:02,017 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:30:02,028 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:30:02,028 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694104201.9713445, 'te': 1694104201.9832962, 'initial_status': True} ====================================================================================================  [2023-09-07 16:30:02,030 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:30:02.030698 [2023-09-07 16:30:02,043 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:30:02,044 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:30:07,529 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104200, schain_finish_ts: 1693742124 [2023-09-07 16:30:08,661 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:30:08,661 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:30:08,661 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:40 - Task submissive-well-off-rotanev-config finished with None [2023-09-07 16:30:08,662 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:42 - Running task submissive-well-off-rotanev-config [2023-09-07 16:30:08,662 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 16:30:09,425 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104200, schain_finish_ts: 1693412520 [2023-09-07 16:30:10,096 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104200, schain_finish_ts: 1692377364 [2023-09-07 16:30:10,432 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': False} [2023-09-07 16:30:10,432 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:30:10,434 INFO][24912][submissive-well-off-rotanev][T_0] - web.models.schain:152 - Changing monitor_last_seen for submissive-well-off-rotanev to 2023-09-07 16:30:10.434531 [2023-09-07 16:30:10,442 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.action:190 - Creating new upstream_config rotation_id: 8, stream: regression-2.2 [2023-09-07 16:30:10,445 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.config.main:49 - Generating sChain config for submissive-well-off-rotanev [2023-09-07 16:30:10,615 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:30:10,615 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:30:10,615 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:30:10,615 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:30:10,616 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:30:10,616 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:30:10,622 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:30:10,627 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 16:30:10,650 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:30:10,954 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:11,241 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:30:11,241 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:30:11,242 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:30:11,242 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:30:11,242 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:30:11,243 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:30:11,246 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:30:11,248 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 16:30:11,283 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:30:11,310 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:11,458 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:30:11,460 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2450 [2023-09-07 16:30:11,460 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:30:11,461 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:30:11,461 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:30:11,461 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:30:11,462 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:30:11,469 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:30:11,472 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:30:11.472258 [2023-09-07 16:30:11,663 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:11,932 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:11,958 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:30:11,989 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:30:11,991 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:30:11,999 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:30:11,999 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694104211.956517, 'te': 1694104211.963205, 'initial_status': True} ====================================================================================================  [2023-09-07 16:30:12,001 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:30:12.001554 [2023-09-07 16:30:12,005 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:30:12,006 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:30:12,113 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:12,222 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:30:12,223 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2435 [2023-09-07 16:30:12,224 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:30:12,224 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:30:12,225 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:30:12,225 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:30:12,225 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:30:12,233 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:30:12,235 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:30:12.235946 [2023-09-07 16:30:12,438 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:12,466 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:30:12,485 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:30:12,486 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:30:12,488 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:30:12.488829 [2023-09-07 16:30:12,495 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:30:12,495 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:30:12,500 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:30:12,502 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:30:12,506 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694104212.494153, 'te': 1694104212.5064406, 'initial_status': True} ====================================================================================================  [2023-09-07 16:30:12,509 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:30:12.509140 [2023-09-07 16:30:12,509 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:30:12,513 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:30:12,514 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:30:12,525 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:30:12,526 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694104212.4651606, 'te': 1694104212.4718676, 'initial_status': True} ====================================================================================================  [2023-09-07 16:30:12,528 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:30:12.528741 [2023-09-07 16:30:12,533 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:30:12,534 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:30:13,094 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:30:13,095 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:30:13,097 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:30:13.097621 [2023-09-07 16:30:13,106 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:30:13,106 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:30:13,120 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:30:13,131 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694104213.1041164, 'te': 1694104213.131139, 'initial_status': True} ====================================================================================================  [2023-09-07 16:30:13,134 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:30:13.134198 [2023-09-07 16:30:13,141 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:30:13,141 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:30:21,153 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104212, schain_finish_ts: 1692637668 [2023-09-07 16:30:22,277 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:30:22,277 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:30:22,277 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:30:22,278 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:30:22,278 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:30:22,278 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:30:22,279 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:30:22,282 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 16:30:22,304 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:30:22,527 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:22,707 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:22,819 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:30:22,826 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2451 [2023-09-07 16:30:22,827 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:30:22,827 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:30:22,828 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:30:22,828 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:30:22,828 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:30:22,834 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:30:22,836 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:30:22.836720 [2023-09-07 16:30:23,073 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:23,106 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:30:23,132 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:30:23,134 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:30:23,140 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:30:23,141 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694104223.104176, 'te': 1694104223.109781, 'initial_status': True} ====================================================================================================  [2023-09-07 16:30:23,142 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:30:23.142966 [2023-09-07 16:30:23,147 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:30:23,148 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:30:24,066 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:30:24,066 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:30:24,069 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:30:24.069235 [2023-09-07 16:30:24,077 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:30:24,077 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:30:24,088 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:30:24,100 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694104224.0753117, 'te': 1694104224.1004012, 'initial_status': True} ====================================================================================================  [2023-09-07 16:30:24,103 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:30:24.103481 [2023-09-07 16:30:24,114 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:30:24,114 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:30:29,939 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104224, schain_finish_ts: 1693742304 [2023-09-07 16:30:31,061 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:30:31,062 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:30:31,062 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:30:31,062 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 16:30:31,062 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 16:30:31,063 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:30:31,064 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:30:31,065 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 16:30:31,088 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:30:31,324 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:31,547 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:31,689 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:30:31,690 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2458 [2023-09-07 16:30:31,691 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:30:31,691 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:30:31,691 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:30:31,691 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:30:31,692 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:30:31,698 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:30:31,699 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:30:31.699959 [2023-09-07 16:30:31,877 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:31,911 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:30:31,949 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:30:31,951 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:30:31,958 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:30:31,959 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694104231.9049537, 'te': 1694104231.9182692, 'initial_status': True} ====================================================================================================  [2023-09-07 16:30:31,960 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:30:31.960747 [2023-09-07 16:30:31,971 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:30:31,971 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:30:32,605 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104224, schain_finish_ts: 1693742124 [2023-09-07 16:30:32,818 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 16:30:32,818 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:30:32,821 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:30:32.821698 [2023-09-07 16:30:32,828 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 16:30:33,566 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104224, schain_finish_ts: 1693412520 [2023-09-07 16:30:33,742 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:30:33,742 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:30:33,742 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 16:30:34,687 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:30:34,687 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:30:34,688 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:30:34,688 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:30:34,688 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:30:34,688 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:30:34,690 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:30:34,691 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 16:30:34,719 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:30:35,044 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:35,291 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:35,413 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:30:35,415 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2451 [2023-09-07 16:30:35,420 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:30:35,420 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:30:35,420 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:30:35,421 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:30:35,421 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:30:35,427 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:30:35,429 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:30:35.429788 [2023-09-07 16:30:35,607 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:35,636 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:30:35,670 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:30:35,672 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:30:35,680 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:30:35,681 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694104235.633527, 'te': 1694104235.6412232, 'initial_status': True} ====================================================================================================  [2023-09-07 16:30:35,683 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:30:35.683125 [2023-09-07 16:30:35,688 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:30:35,688 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:30:36,470 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:30:36,471 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:30:36,474 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:30:36.474045 [2023-09-07 16:30:36,482 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:30:36,483 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:30:36,488 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:30:36,493 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694104236.48144, 'te': 1694104236.4935396, 'initial_status': True} ====================================================================================================  [2023-09-07 16:30:36,495 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:30:36.495770 [2023-09-07 16:30:36,499 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:30:36,500 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:30:36,689 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104224, schain_finish_ts: 1693742004 [2023-09-07 16:30:37,804 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:30:37,805 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:30:37,805 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:30:37,805 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 16:30:37,805 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 16:30:37,806 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:30:37,807 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:30:37,808 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 16:30:37,833 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:30:38,023 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:38,184 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:38,281 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:30:38,282 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2450 [2023-09-07 16:30:38,283 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:30:38,283 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:30:38,284 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:30:38,284 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:30:38,284 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:30:38,292 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:30:38,294 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:30:38.294622 [2023-09-07 16:30:38,465 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:38,494 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:30:38,521 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:30:38,524 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:30:38,536 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:30:38,536 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694104238.4919946, 'te': 1694104238.499254, 'initial_status': True} ====================================================================================================  [2023-09-07 16:30:38,538 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:30:38.538822 [2023-09-07 16:30:38,543 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:30:38,544 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:30:39,551 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 16:30:39,551 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:30:39,554 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:30:39.554412 [2023-09-07 16:30:39,559 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 16:30:47,174 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104236, schain_finish_ts: 1692377364 [2023-09-07 16:30:48,280 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:30:48,280 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:30:48,280 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:30:48,281 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:30:48,281 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:30:48,281 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:30:48,282 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:30:48,283 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 16:30:48,309 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:30:48,500 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:48,691 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:48,833 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:30:48,834 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2436 [2023-09-07 16:30:48,836 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:30:48,836 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:30:48,836 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:30:48,837 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:30:48,837 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:30:48,857 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:30:48,862 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:30:48.862255 [2023-09-07 16:30:49,158 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:49,191 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:30:49,225 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:30:49,227 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:30:49,234 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:30:49,235 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694104249.1893442, 'te': 1694104249.1982381, 'initial_status': True} ====================================================================================================  [2023-09-07 16:30:49,236 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:30:49.236949 [2023-09-07 16:30:49,241 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:30:49,241 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:30:50,129 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:30:50,130 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:30:50,136 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:30:50.136897 [2023-09-07 16:30:50,146 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:30:50,147 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:30:50,160 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:30:50,168 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694104250.145202, 'te': 1694104250.1686459, 'initial_status': True} ====================================================================================================  [2023-09-07 16:30:50,171 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:30:50.171318 [2023-09-07 16:30:50,177 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:30:50,177 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:30:51,544 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 16:30:51,544 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 16:30:53,232 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104248, schain_finish_ts: 1692637668 [2023-09-07 16:30:54,366 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:30:54,366 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:30:54,366 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:30:54,367 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:30:54,367 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:30:54,367 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:30:54,369 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:30:54,370 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 16:30:54,433 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:30:54,624 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:54,786 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:54,884 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:30:54,885 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2452 [2023-09-07 16:30:54,885 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:30:54,886 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:30:54,886 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:30:54,886 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:30:54,886 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:30:54,892 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:30:54,895 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:30:54.895730 [2023-09-07 16:30:55,173 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:30:55,198 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:30:55,225 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:30:55,227 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:30:55,234 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:30:55,234 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694104255.196661, 'te': 1694104255.2038267, 'initial_status': True} ====================================================================================================  [2023-09-07 16:30:55,237 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:30:55.237261 [2023-09-07 16:30:55,242 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:30:55,243 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:30:56,209 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:30:56,209 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:30:56,211 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:30:56.211336 [2023-09-07 16:30:56,220 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:30:56,220 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:30:56,227 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:30:56,234 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694104256.2187824, 'te': 1694104256.2347102, 'initial_status': True} ====================================================================================================  [2023-09-07 16:30:56,236 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:30:56.236878 [2023-09-07 16:30:56,241 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:30:56,242 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:30:58,464 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 16:30:58,465 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 16:30:58,747 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104248, schain_finish_ts: 1693742004 [2023-09-07 16:30:59,881 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:30:59,881 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:30:59,882 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:30:59,884 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:30:59,885 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:30:59,886 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 16:30:59,913 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:31:00,290 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:00,458 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:00,583 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:31:00,585 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2451 [2023-09-07 16:31:00,586 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:31:00,586 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:31:00,587 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:31:00,587 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:31:00,588 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:31:00,597 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:31:00,600 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:31:00.600486 [2023-09-07 16:31:00,812 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:00,839 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:31:00,873 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:31:00,876 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:31:00,885 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:31:00,885 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694104260.8370998, 'te': 1694104260.8450258, 'initial_status': True} ====================================================================================================  [2023-09-07 16:31:00,888 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:31:00.887985 [2023-09-07 16:31:00,893 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:31:00,894 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:31:05,616 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104260, schain_finish_ts: 1693412520 [2023-09-07 16:31:06,742 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:31:06,743 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:31:06,743 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:31:06,743 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:31:06,743 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:31:06,743 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:31:06,745 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:31:06,747 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 16:31:06,770 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:31:07,104 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:07,247 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:07,350 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:31:07,351 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2452 [2023-09-07 16:31:07,352 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:31:07,352 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:31:07,352 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:31:07,353 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:31:07,353 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:31:07,359 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:31:07,361 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:31:07.361085 [2023-09-07 16:31:07,537 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:07,574 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:31:07,616 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:31:07,619 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:31:07,631 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:31:07,632 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694104267.5709858, 'te': 1694104267.5802693, 'initial_status': True} ====================================================================================================  [2023-09-07 16:31:07,635 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:31:07.635800 [2023-09-07 16:31:07,642 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:31:07,643 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:31:08,572 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:31:08,572 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:31:08,576 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:31:08.576256 [2023-09-07 16:31:08,585 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:31:08,585 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:31:08,591 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:31:08,596 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694104268.5839021, 'te': 1694104268.5958526, 'initial_status': True} ====================================================================================================  [2023-09-07 16:31:08,597 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:31:08.597717 [2023-09-07 16:31:08,602 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:31:08,603 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:31:11,585 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 16:31:11,761 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 16:31:11,761 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 16:31:11,762 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 16:31:11,762 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 16:31:11,940 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 16:31:11,941 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 16:31:11,949 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 16:31:11,957 ERROR][11][wan-tasty-kitalpha][T_1] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 16:31:11,958 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:31:11,994 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104260, schain_finish_ts: 1693742304 [2023-09-07 16:31:12,713 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104260, schain_finish_ts: 1693742124 [2023-09-07 16:31:13,168 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:31:13,169 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:31:13,169 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:31:13,169 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:31:13,169 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:31:13,169 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 16:31:13,193 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:31:13,381 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:13,653 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:13,774 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:31:13,776 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2459 [2023-09-07 16:31:13,776 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:31:13,777 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:31:13,777 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:31:13,777 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:31:13,778 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:31:13,784 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:31:13,787 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:31:13.787039 [2023-09-07 16:31:13,871 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:31:13,874 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:31:13,874 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 16:31:13,963 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:13,999 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:31:14,041 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:31:14,044 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:31:14,060 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:31:14,060 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694104273.99771, 'te': 1694104274.0050402, 'initial_status': True} ====================================================================================================  [2023-09-07 16:31:14,062 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:31:14.062914 [2023-09-07 16:31:14,068 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:31:14,068 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:31:18,886 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 16:31:19,067 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 16:31:19,067 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 16:31:19,067 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 16:31:19,068 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 16:31:19,233 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 16:31:19,234 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 16:31:19,236 INFO][8][curly-red-alterf][T_1] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 16:31:19,241 ERROR][8][curly-red-alterf][T_1] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 16:31:19,242 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:31:21,836 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104272, schain_finish_ts: 1693742004 [2023-09-07 16:31:22,985 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:31:22,986 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:31:22,986 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:31:22,987 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 16:31:22,987 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 16:31:22,987 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:31:22,993 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:31:22,994 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 16:31:23,025 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:31:23,312 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:23,555 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:23,749 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:31:23,755 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2452 [2023-09-07 16:31:23,756 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:31:23,756 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:31:23,756 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:31:23,757 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:31:23,758 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:31:23,765 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:31:23,768 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:31:23.768680 [2023-09-07 16:31:24,124 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:24,163 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:31:24,224 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:31:24,226 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:31:24,234 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:31:24,234 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694104284.1611702, 'te': 1694104284.1690137, 'initial_status': True} ====================================================================================================  [2023-09-07 16:31:24,236 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:31:24.236413 [2023-09-07 16:31:24,240 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:31:24,241 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:31:24,817 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 16:31:24,818 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:31:24,821 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:31:24.821581 [2023-09-07 16:31:24,827 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 16:31:26,166 INFO][1][MainProcess][MainThread] - core.schains.process_manager:57 - Process manager started [2023-09-07 16:31:26,386 INFO][1][MainProcess][MainThread] - core.schains.notifications:35 - Node account has 406869999689179237 WEI [2023-09-07 16:31:26,387 INFO][1][MainProcess][MainThread] - tools.notifications.messages:188 - Saving new balance state 3002 1 [2023-09-07 16:31:26,388 INFO][1][MainProcess][MainThread] - core.schains.process_manager:95 - Fetching schains to monitor... [2023-09-07 16:31:29,245 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104284, schain_finish_ts: 1692377364 [2023-09-07 16:31:29,304 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104284, schain_finish_ts: 1692637668 [2023-09-07 16:31:30,381 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:31:30,381 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:31:30,382 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:31:30,384 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:31:30,385 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:31:30,386 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:31:30,388 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:31:30,389 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 16:31:30,420 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:31:30,426 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:31:30,427 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:31:30,427 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:31:30,427 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:31:30,427 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:31:30,427 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:31:30,429 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:31:30,432 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 16:31:30,468 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:31:30,689 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:30,895 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:31,098 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:31,230 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:31:31,231 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2437 [2023-09-07 16:31:31,232 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:31:31,232 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:31:31,232 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:31:31,233 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:31:31,233 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:31:31,240 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:31:31,262 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:31:31.262444 [2023-09-07 16:31:31,298 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:31,438 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:31:31,439 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2453 [2023-09-07 16:31:31,440 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:31:31,440 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:31:31,440 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:31:31,440 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:31:31,441 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:31:31,451 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:31:31,455 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:31:31.455305 [2023-09-07 16:31:31,461 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:31,501 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:31:31,547 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:31:31,549 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:31:31,559 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:31:31,559 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694104291.4920228, 'te': 1694104291.508257, 'initial_status': True} ====================================================================================================  [2023-09-07 16:31:31,561 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:31:31.561851 [2023-09-07 16:31:31,566 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:31:31,567 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:31:31,657 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:31,687 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:31:31,716 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:31:31,717 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:31:31,727 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:31:31,727 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694104291.6857111, 'te': 1694104291.6917958, 'initial_status': True} ====================================================================================================  [2023-09-07 16:31:31,729 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:31:31.729190 [2023-09-07 16:31:31,735 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:31:31,736 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:31:32,026 INFO][1][MainProcess][MainThread] - core.schains.process_manager:109 - Get leaving_history for node ... [2023-09-07 16:31:32,215 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:31:32,216 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:31:32,218 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:31:32.218036 [2023-09-07 16:31:32,224 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:31:32,224 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:31:32,226 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:31:32.226192 [2023-09-07 16:31:32,228 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:31:32,228 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:31:32,238 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:31:32,238 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:31:32,239 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:31:32,250 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694104292.2252848, 'te': 1694104292.2506177, 'initial_status': True} ====================================================================================================  [2023-09-07 16:31:32,252 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:31:32.252658 [2023-09-07 16:31:32,256 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:31:32,257 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:31:32,265 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:31:32,271 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694104292.2336786, 'te': 1694104292.271594, 'initial_status': True} ====================================================================================================  [2023-09-07 16:31:32,274 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:31:32.274400 [2023-09-07 16:31:32,279 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:31:32,280 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:31:33,679 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104284, schain_finish_ts: 1693412520 [2023-09-07 16:31:34,073 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104284, schain_finish_ts: 1692377616 [2023-09-07 16:31:34,829 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:31:34,830 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:31:34,830 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:31:34,830 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:31:34,830 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:31:34,830 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:31:34,833 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:31:34,836 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 16:31:34,870 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:31:35,180 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:35,351 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:35,481 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:31:35,484 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2453 [2023-09-07 16:31:35,487 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:31:35,487 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:31:35,487 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:31:35,488 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:31:35,488 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:31:35,501 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:31:35,505 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:31:35.505860 [2023-09-07 16:31:35,727 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:35,757 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:31:35,794 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:31:35,796 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:31:35,803 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:31:35,804 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694104295.7562075, 'te': 1694104295.7627468, 'initial_status': True} ====================================================================================================  [2023-09-07 16:31:35,805 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:31:35.805591 [2023-09-07 16:31:35,810 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:31:35,810 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:31:36,111 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104284, schain_finish_ts: 1693742304 [2023-09-07 16:31:36,132 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104284, schain_finish_ts: 0 [2023-09-07 16:31:36,508 INFO][1][MainProcess][MainThread] - core.schains.process_manager:117 - Got leaving sChains for the node: [] [2023-09-07 16:31:36,509 INFO][1][MainProcess][MainThread] - core.schains.process_manager:101 -  ==================================================================================================== Monitoring sChains Node ID: 16 sChains on node: [{'name': 'portly-passionate-sirius', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666281384, 'startBlock': 7803380, 'deposit': 0, 'index': 1, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x0000017723ca2', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'curly-red-alterf', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1676058936, 'startBlock': 8469646, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000001fe6cdfa', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'howling-hoedus', 'mainnetOwner': '0x1026467C91fdA40aCc7e6935d1E52f67274FD669', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666181400, 'startBlock': 7796660, 'deposit': 0, 'index': 0, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0xe5a600f4c2608', 'multitransactionMode': False, 'thresholdEncryption': False, 'active': True}, {'name': 'green-giddy-denebola', 'mainnetOwner': '0x8e835d5d9E28ffD259E8Fa8988Cacfc333A582B4', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674565164, 'startBlock': 8368049, 'deposit': 0, 'index': 4, 'generation': 1, 'originator': '0xd2001DAb6898127Be2F167B548691C87251D13C3', 'chainId': '0x00000585eb4b1', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'wan-tasty-kitalpha', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674568392, 'startBlock': 8368271, 'deposit': 0, 'index': 5, 'generation': 1, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x00000152e7be9', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'submissive-well-off-rotanev', 'mainnetOwner': '0x487360568fE5Ee4AE5c75782D5F63e9935B0E512', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1692361104, 'startBlock': 9539053, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000003791fa10', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}] Number of sChains on node: 6 Empty sChain structs: 1 ====================================================================================================  [2023-09-07 16:31:36,673 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:31:36,673 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:31:36,675 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:31:36.675795 [2023-09-07 16:31:36,683 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:31:36,683 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:31:36,689 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:31:36,694 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694104296.6814382, 'te': 1694104296.6939533, 'initial_status': True} ====================================================================================================  [2023-09-07 16:31:36,696 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:31:36.696204 [2023-09-07 16:31:36,702 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:31:36,703 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:31:36,891 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:31:36,891 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain portly-passionate-sirius - Process is running: PID = 7 [2023-09-07 16:31:37,254 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:31:37,254 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:31:37,254 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:31:37,255 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 16:31:37,255 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 16:31:37,256 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:31:37,257 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:31:37,258 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 16:31:37,299 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:31:37,321 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:31:37,323 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain curly-red-alterf - Process is running: PID = 8 [2023-09-07 16:31:37,584 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:37,700 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:31:37,701 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain howling-hoedus - Process is running: PID = 9 [2023-09-07 16:31:37,819 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:37,973 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:31:37,975 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2460 [2023-09-07 16:31:37,975 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:31:37,976 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:31:37,976 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:31:37,976 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:31:37,977 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:31:37,983 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:31:37,985 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:31:37.985544 [2023-09-07 16:31:38,084 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:31:38,085 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain green-giddy-denebola - Process is running: PID = 10 [2023-09-07 16:31:38,284 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:38,325 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:31:38,393 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:31:38,402 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:31:38,422 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:31:38,423 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694104298.3217854, 'te': 1694104298.33316, 'initial_status': True} ====================================================================================================  [2023-09-07 16:31:38,426 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:31:38.426452 [2023-09-07 16:31:38,434 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:31:38,436 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:31:38,460 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:31:38,460 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain wan-tasty-kitalpha - Process is running: PID = 11 [2023-09-07 16:31:38,835 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:31:38,836 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain submissive-well-off-rotanev - Process is running: PID = 24912 [2023-09-07 16:31:38,836 INFO][1][MainProcess][MainThread] - core.schains.process_manager:88 - Process manager procedure finished [2023-09-07 16:31:38,836 INFO][1][MainProcess][MainThread] - __main__:65 - Sleeping for 90s after run_process_manager [2023-09-07 16:31:39,108 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 16:31:39,108 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:31:39,111 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:31:39.111801 [2023-09-07 16:31:39,119 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 16:31:39,876 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104296, schain_finish_ts: 1693742124 [2023-09-07 16:31:41,037 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:31:41,038 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:31:41,038 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 16:31:43,952 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 16:31:43,952 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 16:31:55,369 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104308, schain_finish_ts: 1692637668 [2023-09-07 16:31:56,484 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:31:56,484 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:31:56,484 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:31:56,485 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:31:56,485 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:31:56,485 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:31:56,487 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:31:56,487 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 16:31:56,516 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:31:56,734 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:56,939 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104308, schain_finish_ts: 1693742004 [2023-09-07 16:31:57,083 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:57,213 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:31:57,215 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2454 [2023-09-07 16:31:57,215 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:31:57,216 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:31:57,216 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:31:57,217 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:31:57,217 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:31:57,224 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:31:57,226 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:31:57.226640 [2023-09-07 16:31:57,406 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:57,447 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:31:57,479 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:31:57,482 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:31:57,489 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:31:57,489 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694104317.4455435, 'te': 1694104317.4525185, 'initial_status': True} ====================================================================================================  [2023-09-07 16:31:57,491 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:31:57.491270 [2023-09-07 16:31:57,495 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:31:57,496 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:31:58,099 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:31:58,099 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:31:58,100 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:31:58,100 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:31:58,100 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:31:58,100 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 16:31:58,121 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:31:58,334 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:31:58,335 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:31:58,337 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:31:58.337498 [2023-09-07 16:31:58,342 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 16:31:58,343 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 16:31:58,346 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:31:58,346 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:31:58,354 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:31:58,359 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694104318.344757, 'te': 1694104318.359432, 'initial_status': True} ====================================================================================================  [2023-09-07 16:31:58,362 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:31:58.362585 [2023-09-07 16:31:58,367 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:31:58,367 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:31:58,393 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:58,543 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:58,722 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:31:58,723 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2453 [2023-09-07 16:31:58,724 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:31:58,725 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:31:58,725 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:31:58,726 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:31:58,727 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:31:58,748 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:31:58,751 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:31:58.751572 [2023-09-07 16:31:58,926 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:31:58,955 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:31:58,985 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:31:58,987 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:31:58,994 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:31:58,995 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694104318.9533868, 'te': 1694104318.961567, 'initial_status': True} ====================================================================================================  [2023-09-07 16:31:58,996 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:31:58.996362 [2023-09-07 16:31:59,000 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:31:59,001 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:32:03,092 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:40 - Collecting rotation history for submissive-well-off-rotanev... [2023-09-07 16:32:03,449 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 16:32:03,627 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 16:32:03,628 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 16:32:03,628 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 16:32:03,628 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 16:32:03,819 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 16:32:03,820 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 16:32:03,822 INFO][8][curly-red-alterf][T_0] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 16:32:03,827 ERROR][8][curly-red-alterf][T_0] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 16:32:03,828 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:32:04,232 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:50 - Rotation data for submissive-well-off-rotanev: Rotation(leaving_node_id=14, new_node_id=65, freeze_until=1693741836, rotation_counter=8) [2023-09-07 16:32:05,348 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104308, schain_finish_ts: 1692377364 [2023-09-07 16:32:06,479 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:32:06,479 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:32:06,479 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:32:06,480 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:32:06,480 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:32:06,480 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:32:06,483 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:32:06,483 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 16:32:06,525 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:32:06,864 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:07,131 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:07,262 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:32:07,263 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2438 [2023-09-07 16:32:07,264 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:32:07,264 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:32:07,264 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:32:07,265 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:32:07,265 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:32:07,271 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:32:07,273 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:32:07.273749 [2023-09-07 16:32:07,543 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:07,567 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:32:07,605 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:32:07,607 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:32:07,614 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:32:07,614 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694104327.5655823, 'te': 1694104327.5729864, 'initial_status': True} ====================================================================================================  [2023-09-07 16:32:07,616 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:32:07.616387 [2023-09-07 16:32:07,621 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:32:07,622 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:32:08,295 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:32:08,295 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:32:08,297 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:32:08.297052 [2023-09-07 16:32:08,303 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:32:08,303 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:32:08,310 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:32:08,314 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694104328.3024957, 'te': 1694104328.314152, 'initial_status': True} ====================================================================================================  [2023-09-07 16:32:08,315 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:32:08.315905 [2023-09-07 16:32:08,321 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:32:08,322 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:32:09,755 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104308, schain_finish_ts: 1693412520 [2023-09-07 16:32:10,884 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:32:10,884 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:32:10,884 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:32:10,884 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:32:10,884 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:32:10,884 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:32:10,886 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:32:10,886 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 16:32:10,920 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:32:11,346 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:11,613 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:11,758 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:32:11,761 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2454 [2023-09-07 16:32:11,762 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:32:11,762 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:32:11,763 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:32:11,763 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:32:11,764 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:32:11,772 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:32:11,774 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:32:11.774267 [2023-09-07 16:32:12,015 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:12,045 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:32:12,091 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:32:12,093 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:32:12,102 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:32:12,103 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694104332.0423834, 'te': 1694104332.0520422, 'initial_status': True} ====================================================================================================  [2023-09-07 16:32:12,111 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:32:12.111178 [2023-09-07 16:32:12,120 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:32:12,137 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:32:12,792 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:32:12,793 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:32:12,795 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:32:12.795740 [2023-09-07 16:32:12,802 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:32:12,803 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:32:12,808 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:32:12,814 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694104332.8014252, 'te': 1694104332.8141344, 'initial_status': True} ====================================================================================================  [2023-09-07 16:32:12,815 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:32:12.815954 [2023-09-07 16:32:12,819 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:32:12,820 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:32:13,217 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104308, schain_finish_ts: 1693742304 [2023-09-07 16:32:14,356 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:32:14,356 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:32:14,356 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:32:14,356 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:32:14,357 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:32:14,357 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 16:32:14,377 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:32:14,639 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:15,015 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:15,131 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:32:15,137 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2461 [2023-09-07 16:32:15,142 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:32:15,142 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:32:15,142 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:32:15,143 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:32:15,143 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:32:15,151 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:32:15,157 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:32:15.157393 [2023-09-07 16:32:15,369 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:15,396 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:32:15,424 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:32:15,426 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:32:15,433 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:32:15,433 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694104335.3938615, 'te': 1694104335.4004796, 'initial_status': True} ====================================================================================================  [2023-09-07 16:32:15,435 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:32:15.435372 [2023-09-07 16:32:15,440 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:32:15,441 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:32:18,073 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 16:32:18,259 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 16:32:18,260 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 16:32:18,261 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 16:32:18,261 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 16:32:18,442 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 16:32:18,443 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 16:32:18,445 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 16:32:18,449 ERROR][11][wan-tasty-kitalpha][T_1] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 16:32:18,450 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:32:19,681 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 63 -> 65 [2023-09-07 16:32:19,939 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104332, schain_finish_ts: 1693742124 [2023-09-07 16:32:21,044 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:32:21,049 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:32:21,051 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 16:32:24,438 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104332, schain_finish_ts: 1692637668 [2023-09-07 16:32:24,671 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 23 -> 24 [2023-09-07 16:32:25,602 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:32:25,602 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:32:25,602 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:32:25,603 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:32:25,603 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:32:25,603 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:32:25,606 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:32:25,609 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 16:32:25,636 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:32:25,883 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:26,111 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:26,230 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:32:26,231 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2455 [2023-09-07 16:32:26,232 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:32:26,232 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:32:26,233 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:32:26,233 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:32:26,233 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:32:26,240 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:32:26,242 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:32:26.242283 [2023-09-07 16:32:26,415 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:26,440 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:32:26,469 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:32:26,471 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:32:26,478 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:32:26,479 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694104346.4385293, 'te': 1694104346.4447823, 'initial_status': True} ====================================================================================================  [2023-09-07 16:32:26,480 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:32:26.480897 [2023-09-07 16:32:26,486 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:32:26,490 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:32:27,407 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:32:27,407 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:32:27,412 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:32:27.411985 [2023-09-07 16:32:27,420 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:32:27,420 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:32:27,425 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:32:27,430 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694104347.4186027, 'te': 1694104347.4301782, 'initial_status': True} ====================================================================================================  [2023-09-07 16:32:27,432 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:32:27.432407 [2023-09-07 16:32:27,437 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:32:27,437 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:32:29,227 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 9 -> 64 [2023-09-07 16:32:31,045 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104344, schain_finish_ts: 1693742004 [2023-09-07 16:32:32,180 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:32:32,181 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:32:32,183 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:32:32,183 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 16:32:32,184 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 16:32:32,186 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:32:32,189 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:32:32,193 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 16:32:32,232 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:32:32,458 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:32,618 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:32,722 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:32:32,724 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2454 [2023-09-07 16:32:32,726 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:32:32,726 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:32:32,727 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:32:32,727 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:32:32,728 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:32:32,740 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:32:32,745 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:32:32.745013 [2023-09-07 16:32:33,068 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:33,094 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:32:33,125 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:32:33,127 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:32:33,135 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:32:33,135 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694104353.0925946, 'te': 1694104353.0994105, 'initial_status': True} ====================================================================================================  [2023-09-07 16:32:33,137 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:32:33.137192 [2023-09-07 16:32:33,142 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:32:33,145 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:32:33,421 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104344, schain_finish_ts: 1692377364 [2023-09-07 16:32:34,047 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 16:32:34,047 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:32:34,051 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:32:34.051181 [2023-09-07 16:32:34,058 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 16:32:34,206 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 35 -> 16 [2023-09-07 16:32:34,572 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:32:34,573 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:32:34,573 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:32:34,573 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:32:34,573 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:32:34,574 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:32:34,575 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:32:34,575 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 16:32:34,601 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:32:34,838 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104344, schain_finish_ts: 1693412520 [2023-09-07 16:32:34,943 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:35,215 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:35,357 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:32:35,358 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2439 [2023-09-07 16:32:35,358 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:32:35,359 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:32:35,359 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:32:35,359 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:32:35,360 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:32:35,366 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:32:35,368 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:32:35.367990 [2023-09-07 16:32:35,518 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:35,545 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:32:35,599 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:32:35,602 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:32:35,612 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:32:35,612 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694104355.5428195, 'te': 1694104355.5623336, 'initial_status': True} ====================================================================================================  [2023-09-07 16:32:35,615 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:32:35.614982 [2023-09-07 16:32:35,620 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:32:35,620 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:32:35,983 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:32:35,983 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:32:35,983 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:32:35,983 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:32:35,984 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:32:35,984 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:32:35,985 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:32:35,986 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 16:32:36,014 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:32:36,267 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:36,395 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:32:36,396 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:32:36,399 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:32:36.399530 [2023-09-07 16:32:36,408 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:32:36,408 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:32:36,416 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:32:36,422 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694104356.4058, 'te': 1694104356.4219496, 'initial_status': True} ====================================================================================================  [2023-09-07 16:32:36,425 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:32:36.425139 [2023-09-07 16:32:36,430 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:32:36,430 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:32:36,451 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:36,589 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:32:36,601 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2455 [2023-09-07 16:32:36,602 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:32:36,602 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:32:36,603 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:32:36,603 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:32:36,603 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:32:36,610 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:32:36,612 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:32:36.612965 [2023-09-07 16:32:36,954 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:36,986 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:32:37,030 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:32:37,033 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:32:37,044 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:32:37,044 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694104356.9842703, 'te': 1694104356.99445, 'initial_status': True} ====================================================================================================  [2023-09-07 16:32:37,047 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:32:37.047247 [2023-09-07 16:32:37,053 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:32:37,054 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:32:37,882 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:32:37,882 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:32:37,884 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:32:37.884738 [2023-09-07 16:32:37,892 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:32:37,892 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:32:37,898 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:32:37,903 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694104357.890639, 'te': 1694104357.903495, 'initial_status': True} ====================================================================================================  [2023-09-07 16:32:37,906 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:32:37.906061 [2023-09-07 16:32:37,913 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:32:37,913 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:32:39,571 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 55 -> 63 [2023-09-07 16:32:45,315 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 14 -> 11 [2023-09-07 16:32:47,560 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104356, schain_finish_ts: 1692637668 [2023-09-07 16:32:48,711 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:32:48,711 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:32:48,711 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:32:48,712 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:32:48,712 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:32:48,712 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:32:48,714 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:32:48,716 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 16:32:48,746 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:32:48,942 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:49,176 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:49,271 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:32:49,271 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2456 [2023-09-07 16:32:49,272 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:32:49,272 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:32:49,272 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:32:49,273 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:32:49,273 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:32:49,278 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:32:49,280 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:32:49.280557 [2023-09-07 16:32:49,412 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:49,437 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:32:49,465 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:32:49,467 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:32:49,475 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:32:49,475 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694104369.4356837, 'te': 1694104369.4424226, 'initial_status': True} ====================================================================================================  [2023-09-07 16:32:49,476 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:32:49.476880 [2023-09-07 16:32:49,482 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:32:49,483 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:32:49,949 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104368, schain_finish_ts: 1693742124 [2023-09-07 16:32:51,075 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:32:51,075 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:32:51,075 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 16:32:51,077 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:32:51,082 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:32:51,086 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:32:51.086773 [2023-09-07 16:32:51,096 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:32:51,097 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:32:51,103 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:32:51,108 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694104371.0939586, 'te': 1694104371.1077695, 'initial_status': True} ====================================================================================================  [2023-09-07 16:32:51,109 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:32:51.109609 [2023-09-07 16:32:51,114 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:32:51,114 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:32:52,314 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104368, schain_finish_ts: 1693742304 [2023-09-07 16:32:52,541 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 56 -> 60 [2023-09-07 16:32:53,446 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:32:53,447 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:32:53,447 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:32:53,447 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 16:32:53,447 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 16:32:53,447 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:32:53,450 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:32:53,451 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 16:32:53,477 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:32:53,669 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:53,829 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:53,918 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 16:32:53,918 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 16:32:53,959 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:32:53,960 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2462 [2023-09-07 16:32:53,961 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:32:53,961 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:32:53,962 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:32:53,962 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:32:53,962 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:32:53,969 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:32:53,972 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:32:53.972174 [2023-09-07 16:32:54,187 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:54,216 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:32:54,243 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:32:54,245 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:32:54,252 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:32:54,253 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694104374.2149708, 'te': 1694104374.2218049, 'initial_status': True} ====================================================================================================  [2023-09-07 16:32:54,254 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:32:54.254773 [2023-09-07 16:32:54,260 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:32:54,260 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:32:55,223 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 16:32:55,223 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:32:55,225 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:32:55.225532 [2023-09-07 16:32:55,231 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 16:32:58,125 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104368, schain_finish_ts: 1693742004 [2023-09-07 16:32:58,472 ERROR][24912][submissive-well-off-rotanev][T_0] - core.schains.task:26 - Task submissive-well-off-rotanev-config failed with max() arg is an empty sequence Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 63, in execute self.am.upstream_config() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 194, in upstream_config new_config = create_new_upstream_config( File "/usr/src/admin/core/schains/config/main.py", line 51, in create_new_upstream_config schain_config = generate_schain_config_with_skale( File "/usr/src/admin/core/schains/config/generator.py", line 213, in generate_schain_config_with_skale node_groups = get_previous_schain_groups(skale, schain_name) File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 56, in get_previous_schain_groups _add_previous_schain_rotations_state( File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 119, in _add_previous_schain_rotations_state new_node_id = max(previous_nodes.items(), key=lambda x: x[1]['finish_ts'])[0] ValueError: max() arg is an empty sequence [2023-09-07 16:32:58,472 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:32:59,234 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:32:59,235 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:32:59,235 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:32:59,235 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:32:59,235 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:32:59,236 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 16:32:59,269 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:32:59,562 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:59,773 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:32:59,917 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:32:59,918 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2455 [2023-09-07 16:32:59,919 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:32:59,919 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:32:59,920 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:32:59,920 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:32:59,921 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:32:59,931 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:32:59,934 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:32:59.934528 [2023-09-07 16:33:00,170 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:00,202 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:33:00,247 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:33:00,254 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:33:00,261 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:33:00,262 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694104380.2004507, 'te': 1694104380.2078314, 'initial_status': True} ====================================================================================================  [2023-09-07 16:33:00,264 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:33:00.263992 [2023-09-07 16:33:00,270 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:33:00,271 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:33:00,530 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104368, schain_finish_ts: 1692377364 [2023-09-07 16:33:01,662 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:33:01,664 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:33:01,666 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:33:01,670 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:33:01,670 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:33:01,670 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:33:01,672 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:33:01,672 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 16:33:01,702 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:33:01,905 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:02,117 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:02,228 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:33:02,229 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2440 [2023-09-07 16:33:02,229 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:33:02,230 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:33:02,230 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:33:02,230 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:33:02,230 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:33:02,249 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:33:02,253 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:33:02.253648 [2023-09-07 16:33:02,555 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:02,589 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:33:02,624 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:33:02,626 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:33:02,633 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:33:02,634 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694104382.5871768, 'te': 1694104382.5974205, 'initial_status': True} ====================================================================================================  [2023-09-07 16:33:02,635 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:33:02.635426 [2023-09-07 16:33:02,644 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:33:02,645 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:33:02,919 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104380, schain_finish_ts: 1693412520 [2023-09-07 16:33:03,462 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:33:03,462 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:33:03,465 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:33:03.465721 [2023-09-07 16:33:03,472 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:33:03,473 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:33:03,478 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:33:03,485 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694104383.4713914, 'te': 1694104383.4847846, 'initial_status': True} ====================================================================================================  [2023-09-07 16:33:03,486 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:33:03.486697 [2023-09-07 16:33:03,490 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:33:03,491 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:33:04,054 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:33:04,054 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:33:04,054 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:33:04,055 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:33:04,055 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:33:04,055 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:33:04,057 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:33:04,061 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 16:33:04,091 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:33:04,314 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:04,463 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:04,560 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:33:04,561 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2456 [2023-09-07 16:33:04,561 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:33:04,562 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:33:04,562 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:33:04,562 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:33:04,562 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:33:04,568 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:33:04,570 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:33:04.570303 [2023-09-07 16:33:04,739 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:04,773 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:33:04,809 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:33:04,812 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:33:04,819 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:33:04,820 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694104384.7662299, 'te': 1694104384.7817502, 'initial_status': True} ====================================================================================================  [2023-09-07 16:33:04,822 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:33:04.822414 [2023-09-07 16:33:04,828 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:33:04,829 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:33:05,854 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:33:05,854 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:33:05,856 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:33:05.856141 [2023-09-07 16:33:05,863 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:33:05,863 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:33:05,869 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:33:05,875 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694104385.8617327, 'te': 1694104385.8747663, 'initial_status': True} ====================================================================================================  [2023-09-07 16:33:05,876 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:33:05.876759 [2023-09-07 16:33:05,881 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:33:05,881 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:33:08,929 INFO][1][MainProcess][MainThread] - core.schains.cleaner:65 - Cleaner process started [2023-09-07 16:33:08,963 INFO][30390][cleaner][MainThread] - core.schains.cleaner:110 - Cleaner procedure started. [2023-09-07 16:33:12,047 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104380, schain_finish_ts: 1693742124 [2023-09-07 16:33:13,187 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:33:13,188 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:33:13,188 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:40 - Task submissive-well-off-rotanev-config finished with None [2023-09-07 16:33:13,188 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:42 - Running task submissive-well-off-rotanev-config [2023-09-07 16:33:13,189 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 16:33:13,204 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 16:33:13,381 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 16:33:13,382 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 16:33:13,382 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 16:33:13,382 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 16:33:13,571 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 16:33:13,571 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 16:33:13,573 INFO][8][curly-red-alterf][T_1] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 16:33:13,583 ERROR][8][curly-red-alterf][T_1] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 16:33:13,584 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:33:13,661 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104380, schain_finish_ts: 1692637668 [2023-09-07 16:33:14,246 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 16:33:14,246 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 16:33:14,580 INFO][30390][cleaner][MainThread] - core.schains.cleaner:116 - sChains on contracts: ['portly-passionate-sirius', 'curly-red-alterf', 'howling-hoedus', 'green-giddy-denebola', 'wan-tasty-kitalpha', '', 'submissive-well-off-rotanev'] sChains on node: ['curly-red-alterf', 'green-giddy-denebola', 'howling-hoedus', 'portly-passionate-sirius', 'submissive-well-off-rotanev', 'wan-tasty-kitalpha'] [2023-09-07 16:33:14,580 INFO][30390][cleaner][MainThread] - core.schains.cleaner:132 - Cleanup procedure finished [2023-09-07 16:33:14,584 INFO][1][MainProcess][MainThread] - core.schains.cleaner:67 - Cleaner process is joined. [2023-09-07 16:33:14,585 INFO][1][MainProcess][MainThread] - core.schains.cleaner:68 - Terminating the process [2023-09-07 16:33:14,585 INFO][1][MainProcess][MainThread] - __main__:70 - Sleeping for 90s after run_cleaner [2023-09-07 16:33:14,799 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:33:14,799 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:33:14,799 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:33:14,800 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:33:14,800 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:33:14,800 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:33:14,802 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:33:14,803 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 16:33:14,827 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:33:14,920 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': False} [2023-09-07 16:33:14,921 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:33:14,924 INFO][24912][submissive-well-off-rotanev][T_0] - web.models.schain:152 - Changing monitor_last_seen for submissive-well-off-rotanev to 2023-09-07 16:33:14.923995 [2023-09-07 16:33:14,930 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.action:190 - Creating new upstream_config rotation_id: 8, stream: regression-2.2 [2023-09-07 16:33:14,932 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.config.main:49 - Generating sChain config for submissive-well-off-rotanev [2023-09-07 16:33:15,102 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:15,280 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:15,388 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:33:15,389 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2457 [2023-09-07 16:33:15,390 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:33:15,390 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:33:15,390 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:33:15,390 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:33:15,391 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:33:15,397 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:33:15,399 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:33:15.399109 [2023-09-07 16:33:15,565 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:15,642 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:33:15,678 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:33:15,681 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:33:15,690 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:33:15,691 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694104395.6410081, 'te': 1694104395.647445, 'initial_status': True} ====================================================================================================  [2023-09-07 16:33:15,693 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:33:15.693742 [2023-09-07 16:33:15,699 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:33:15,700 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:33:16,587 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:33:16,588 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:33:16,590 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:33:16.590174 [2023-09-07 16:33:16,597 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:33:16,598 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:33:16,604 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:33:16,609 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694104396.5962262, 'te': 1694104396.6088073, 'initial_status': True} ====================================================================================================  [2023-09-07 16:33:16,610 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:33:16.610792 [2023-09-07 16:33:16,615 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:33:16,615 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:33:21,170 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104392, schain_finish_ts: 1693742004 [2023-09-07 16:33:22,296 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:33:22,296 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:33:22,296 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:33:22,297 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 16:33:22,297 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 16:33:22,297 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:33:22,300 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:33:22,301 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 16:33:22,336 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:33:22,744 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:23,097 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:23,219 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:33:23,220 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2456 [2023-09-07 16:33:23,221 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:33:23,221 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:33:23,222 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:33:23,222 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:33:23,223 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:33:23,230 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:33:23,232 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:33:23.232904 [2023-09-07 16:33:23,402 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:23,426 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:33:23,454 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:33:23,456 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:33:23,464 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:33:23,464 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694104403.4251902, 'te': 1694104403.4314606, 'initial_status': True} ====================================================================================================  [2023-09-07 16:33:23,467 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:33:23.467661 [2023-09-07 16:33:23,472 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:33:23,473 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:33:24,230 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 16:33:24,230 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:33:24,232 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:33:24.232817 [2023-09-07 16:33:24,237 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 16:33:28,393 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104404, schain_finish_ts: 1693742304 [2023-09-07 16:33:28,995 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104404, schain_finish_ts: 1693412520 [2023-09-07 16:33:29,530 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:33:29,531 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:33:29,531 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:33:29,531 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:33:29,531 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:33:29,531 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 16:33:29,553 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:33:29,689 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104404, schain_finish_ts: 1692377364 [2023-09-07 16:33:29,967 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:30,135 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:33:30,136 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:33:30,136 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:33:30,136 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:33:30,137 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:33:30,137 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:33:30,141 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:33:30,145 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 16:33:30,170 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:33:30,190 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:30,339 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:33:30,340 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2463 [2023-09-07 16:33:30,344 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:33:30,344 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:33:30,345 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:33:30,345 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:33:30,347 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:33:30,355 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:33:30,359 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:33:30.358993 [2023-09-07 16:33:30,411 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:30,617 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:30,651 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:33:30,698 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:33:30,700 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:33:30,708 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:33:30,708 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694104410.64984, 'te': 1694104410.6563895, 'initial_status': True} ====================================================================================================  [2023-09-07 16:33:30,710 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:33:30.710069 [2023-09-07 16:33:30,715 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:33:30,716 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:33:30,902 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:33:30,902 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:33:30,903 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:33:30,903 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:33:30,903 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:33:30,903 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:33:30,905 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:33:30,908 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 16:33:30,933 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:33:30,959 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:31,102 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:33:31,103 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2457 [2023-09-07 16:33:31,104 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:33:31,104 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:33:31,105 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:33:31,106 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:33:31,107 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:33:31,125 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:33:31,130 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:33:31.130147 [2023-09-07 16:33:31,170 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:31,337 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:31,379 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:33:31,422 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:33:31,428 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:33:31,444 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:33:31,445 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694104411.3769665, 'te': 1694104411.3867493, 'initial_status': True} ====================================================================================================  [2023-09-07 16:33:31,449 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:33:31.449039 [2023-09-07 16:33:31,454 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:33:31,455 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:33:31,516 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:31,632 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:33:31,633 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2441 [2023-09-07 16:33:31,634 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:33:31,634 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:33:31,634 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:33:31,635 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:33:31,635 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:33:31,641 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:33:31,646 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:33:31.646234 [2023-09-07 16:33:31,805 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:31,834 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:33:31,878 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:33:31,881 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:33:31,889 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:33:31,890 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694104411.8321476, 'te': 1694104411.8423753, 'initial_status': True} ====================================================================================================  [2023-09-07 16:33:31,891 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:33:31.891873 [2023-09-07 16:33:31,898 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:33:31,899 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:33:31,979 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:33:31,980 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:33:31,982 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:33:31.982894 [2023-09-07 16:33:31,996 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:33:31,996 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:33:32,006 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:33:32,012 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694104411.994964, 'te': 1694104412.012149, 'initial_status': True} ====================================================================================================  [2023-09-07 16:33:32,014 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:33:32.014613 [2023-09-07 16:33:32,020 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:33:32,020 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:33:32,682 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:33:32,683 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:33:32,685 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:33:32.685532 [2023-09-07 16:33:32,694 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:33:32,694 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:33:32,700 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:33:32,704 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694104412.6922126, 'te': 1694104412.7042074, 'initial_status': True} ====================================================================================================  [2023-09-07 16:33:32,706 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:33:32.706048 [2023-09-07 16:33:32,711 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:33:32,711 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:33:33,929 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 16:33:34,097 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 16:33:34,098 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 16:33:34,098 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 16:33:34,098 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 16:33:34,259 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 16:33:34,260 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 16:33:34,262 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 16:33:34,267 ERROR][11][wan-tasty-kitalpha][T_0] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 16:33:34,268 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:33:39,142 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104404, schain_finish_ts: 1693742124 [2023-09-07 16:33:40,276 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:33:40,276 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:33:40,276 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 16:33:43,404 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 16:33:43,404 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 16:33:49,782 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104416, schain_finish_ts: 1692637668 [2023-09-07 16:33:50,958 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:33:50,958 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:33:50,958 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:33:50,959 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:33:50,959 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:33:50,959 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:33:50,963 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:33:50,965 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 16:33:50,987 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:33:51,097 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104416, schain_finish_ts: 1693412520 [2023-09-07 16:33:51,323 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:51,496 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:51,601 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:33:51,602 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2458 [2023-09-07 16:33:51,603 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:33:51,603 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:33:51,603 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:33:51,604 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:33:51,604 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:33:51,609 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:33:51,611 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:33:51.611800 [2023-09-07 16:33:51,772 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:51,803 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:33:51,832 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:33:51,834 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:33:51,841 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:33:51,841 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694104431.8007107, 'te': 1694104431.8093398, 'initial_status': True} ====================================================================================================  [2023-09-07 16:33:51,843 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:33:51.843137 [2023-09-07 16:33:51,848 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:33:51,848 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:33:52,254 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:33:52,255 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:33:52,255 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:33:52,255 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:33:52,255 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:33:52,255 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:33:52,257 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:33:52,257 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 16:33:52,283 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:33:52,573 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:52,719 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:52,814 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:33:52,818 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:33:52,824 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:33:52.824632 [2023-09-07 16:33:52,836 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:33:52,836 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:33:52,840 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:33:52,842 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2458 [2023-09-07 16:33:52,844 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:33:52,844 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:33:52,844 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:33:52,844 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:33:52,845 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:33:52,846 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:33:52,851 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:33:52,855 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694104432.835005, 'te': 1694104432.8550587, 'initial_status': True} ====================================================================================================  [2023-09-07 16:33:52,857 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:33:52.857798 [2023-09-07 16:33:52,859 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:33:52.859856 [2023-09-07 16:33:52,871 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:33:52,872 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:33:53,024 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:53,054 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:33:53,083 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:33:53,085 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:33:53,092 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:33:53,093 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694104433.0529435, 'te': 1694104433.0595272, 'initial_status': True} ====================================================================================================  [2023-09-07 16:33:53,095 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:33:53.095008 [2023-09-07 16:33:53,099 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:33:53,100 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:33:54,061 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:33:54,061 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:33:54,063 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:33:54.063781 [2023-09-07 16:33:54,071 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:33:54,072 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:33:54,078 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:33:54,085 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694104434.0704422, 'te': 1694104434.0850801, 'initial_status': True} ====================================================================================================  [2023-09-07 16:33:54,087 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:33:54.087619 [2023-09-07 16:33:54,093 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:33:54,094 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:33:58,272 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104428, schain_finish_ts: 1693742004 [2023-09-07 16:33:59,436 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:33:59,437 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:33:59,437 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:33:59,437 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:33:59,438 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:33:59,438 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 16:33:59,468 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:33:59,783 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:33:59,962 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:34:00,100 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:34:00,104 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2457 [2023-09-07 16:34:00,105 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:34:00,105 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:34:00,105 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:34:00,106 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:34:00,106 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:34:00,113 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:34:00,116 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:34:00.116625 [2023-09-07 16:34:00,336 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:34:00,371 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:34:00,402 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:34:00,406 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:34:00,420 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:34:00,421 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694104440.3696122, 'te': 1694104440.3765175, 'initial_status': True} ====================================================================================================  [2023-09-07 16:34:00,425 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:34:00.425178 [2023-09-07 16:34:00,431 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:34:00,432 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:34:00,489 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104428, schain_finish_ts: 1693742304 [2023-09-07 16:34:01,613 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:34:01,613 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:34:01,614 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:34:01,614 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 16:34:01,614 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 16:34:01,614 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:34:01,616 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:34:01,617 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 16:34:01,639 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:34:01,825 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:34:01,975 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:34:02,082 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:34:02,083 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2464 [2023-09-07 16:34:02,088 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:34:02,088 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:34:02,088 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:34:02,089 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:34:02,089 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:34:02,095 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:34:02,097 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:34:02.097947 [2023-09-07 16:34:02,324 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:34:02,352 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:34:02,389 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:34:02,392 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:34:02,400 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:34:02,401 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694104442.34965, 'te': 1694104442.3568394, 'initial_status': True} ====================================================================================================  [2023-09-07 16:34:02,403 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:34:02.403256 [2023-09-07 16:34:02,407 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:34:02,408 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:34:03,181 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 16:34:03,359 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 16:34:03,359 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 16:34:03,360 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 16:34:03,360 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 16:34:03,373 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 16:34:03,373 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:34:03,375 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:34:03.375623 [2023-09-07 16:34:03,383 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 16:34:03,535 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 16:34:03,536 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 16:34:03,538 INFO][8][curly-red-alterf][T_0] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 16:34:03,546 ERROR][8][curly-red-alterf][T_0] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 16:34:03,546 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:34:09,880 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104440, schain_finish_ts: 1692377364 [2023-09-07 16:34:11,017 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:34:11,017 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:34:11,017 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:34:11,018 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:34:11,018 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:34:11,018 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:34:11,021 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:34:11,022 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 16:34:11,077 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:34:11,201 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104440, schain_finish_ts: 1693742124 [2023-09-07 16:34:11,340 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:34:11,620 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:34:11,754 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:34:11,763 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2442 [2023-09-07 16:34:11,769 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:34:11,769 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:34:11,769 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:34:11,770 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:34:11,770 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:34:11,778 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:34:11,781 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:34:11.781454 [2023-09-07 16:34:11,965 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:34:11,991 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:34:12,019 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:34:12,021 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:34:12,036 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:34:12,039 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694104451.989794, 'te': 1694104451.9963582, 'initial_status': True} ====================================================================================================  [2023-09-07 16:34:12,044 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:34:12.044292 [2023-09-07 16:34:12,050 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:34:12,051 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:34:12,344 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:34:12,344 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:34:12,345 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 16:34:12,831 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:34:12,831 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:34:12,833 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:34:12.833560 [2023-09-07 16:34:12,840 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:34:12,840 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:34:12,846 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:34:12,851 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694104452.8388348, 'te': 1694104452.8508172, 'initial_status': True} ====================================================================================================  [2023-09-07 16:34:12,853 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:34:12.853558 [2023-09-07 16:34:12,860 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:34:12,860 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:34:18,897 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104452, schain_finish_ts: 1692637668 [2023-09-07 16:34:20,063 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:34:20,063 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:34:20,064 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:34:20,064 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:34:20,065 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:34:20,065 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:34:20,069 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:34:20,071 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 16:34:20,109 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:34:20,384 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:34:20,416 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104452, schain_finish_ts: 1693742004 [2023-09-07 16:34:20,611 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:34:20,773 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:34:20,774 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2459 [2023-09-07 16:34:20,776 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:34:20,777 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:34:20,777 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:34:20,779 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:34:20,780 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:34:20,795 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:34:20,799 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:34:20.799313 [2023-09-07 16:34:20,988 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:34:21,030 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:34:21,095 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:34:21,098 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:34:21,106 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:34:21,106 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694104461.024165, 'te': 1694104461.0486119, 'initial_status': True} ====================================================================================================  [2023-09-07 16:34:21,108 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:34:21.108421 [2023-09-07 16:34:21,113 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:34:21,113 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:34:21,575 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:34:21,576 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:34:21,576 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:34:21,576 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 16:34:21,576 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 16:34:21,577 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:34:21,579 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:34:21,580 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 16:34:21,609 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:34:21,863 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:34:21,864 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:34:21,866 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:34:21.866604 [2023-09-07 16:34:21,885 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:34:21,886 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:34:21,897 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:34:21,905 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694104461.8838918, 'te': 1694104461.9055185, 'initial_status': True} ====================================================================================================  [2023-09-07 16:34:21,906 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:34:21,907 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:34:21.907958 [2023-09-07 16:34:21,914 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:34:21,914 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:34:22,257 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:34:22,324 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 16:34:22,324 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 16:34:22,385 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:34:22,388 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2458 [2023-09-07 16:34:22,390 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:34:22,391 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:34:22,391 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:34:22,392 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:34:22,392 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:34:22,405 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:34:22,407 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:34:22.407928 [2023-09-07 16:34:22,639 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:34:22,673 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:34:22,718 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:34:22,721 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:34:22,733 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:34:22,735 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694104462.6710625, 'te': 1694104462.6826613, 'initial_status': True} ====================================================================================================  [2023-09-07 16:34:22,738 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:34:22.738942 [2023-09-07 16:34:22,744 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:34:22,744 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:34:23,411 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 16:34:23,411 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:34:23,413 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:34:23.413910 [2023-09-07 16:34:23,419 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 16:34:26,219 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104452, schain_finish_ts: 1693412520 [2023-09-07 16:34:27,384 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:34:27,385 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:34:27,385 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:34:27,386 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:34:27,387 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:34:27,391 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:34:27,396 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:34:27,396 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 16:34:27,419 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:34:27,718 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:34:27,884 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:34:27,998 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:34:27,999 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2459 [2023-09-07 16:34:28,000 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:34:28,000 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:34:28,000 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:34:28,000 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:34:28,001 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:34:28,007 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:34:28,010 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:34:28.010769 [2023-09-07 16:34:28,184 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:34:28,210 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:34:28,243 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:34:28,245 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:34:28,253 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:34:28,253 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694104468.2087905, 'te': 1694104468.2181458, 'initial_status': True} ====================================================================================================  [2023-09-07 16:34:28,255 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:34:28.255047 [2023-09-07 16:34:28,259 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:34:28,260 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:34:29,244 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:34:29,244 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:34:29,246 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:34:29.246382 [2023-09-07 16:34:29,252 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:34:29,252 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:34:29,258 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:34:29,263 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694104469.251223, 'te': 1694104469.2635121, 'initial_status': True} ====================================================================================================  [2023-09-07 16:34:29,265 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:34:29.265296 [2023-09-07 16:34:29,270 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:34:29,270 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:34:31,602 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104464, schain_finish_ts: 1693742304 [2023-09-07 16:34:32,772 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:34:32,772 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:34:32,772 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:34:32,772 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:34:32,773 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:34:32,773 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 16:34:32,793 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:34:33,128 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:34:33,282 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:34:33,396 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:34:33,400 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2465 [2023-09-07 16:34:33,400 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:34:33,401 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:34:33,401 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:34:33,401 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:34:33,401 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:34:33,407 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:34:33,409 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:34:33.409838 [2023-09-07 16:34:33,552 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:34:33,578 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:34:33,630 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:34:33,633 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:34:33,644 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:34:33,644 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694104473.5765262, 'te': 1694104473.5844023, 'initial_status': True} ====================================================================================================  [2023-09-07 16:34:33,646 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:34:33.646897 [2023-09-07 16:34:33,654 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:34:33,655 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:34:38,294 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104476, schain_finish_ts: 1693742124 [2023-09-07 16:34:39,437 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:34:39,438 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:34:39,438 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 16:34:41,789 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 16:34:41,957 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104476, schain_finish_ts: 1692377364 [2023-09-07 16:34:41,972 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 16:34:41,973 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 16:34:41,973 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 16:34:41,973 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 16:34:42,148 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 16:34:42,149 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 16:34:42,151 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 16:34:42,157 ERROR][11][wan-tasty-kitalpha][T_1] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 16:34:42,157 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:34:42,413 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 16:34:42,413 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 16:34:43,104 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:34:43,104 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:34:43,104 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:34:43,105 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:34:43,105 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:34:43,105 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:34:43,111 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:34:43,112 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 16:34:43,186 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:34:43,441 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:34:43,603 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:34:43,782 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:34:43,785 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2443 [2023-09-07 16:34:43,786 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:34:43,789 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:34:43,789 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:34:43,790 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:34:43,790 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:34:43,824 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:34:43,826 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:34:43.826923 [2023-09-07 16:34:44,189 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:34:44,215 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:34:44,253 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:34:44,257 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:34:44,267 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:34:44,267 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694104484.2136438, 'te': 1694104484.2216866, 'initial_status': True} ====================================================================================================  [2023-09-07 16:34:44,270 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:34:44.270276 [2023-09-07 16:34:44,275 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:34:44,276 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:34:44,675 INFO][1][MainProcess][MainThread] - core.schains.process_manager:57 - Process manager started [2023-09-07 16:34:44,890 INFO][1][MainProcess][MainThread] - core.schains.notifications:35 - Node account has 406869999689179237 WEI [2023-09-07 16:34:44,891 INFO][1][MainProcess][MainThread] - tools.notifications.messages:188 - Saving new balance state 3003 1 [2023-09-07 16:34:44,891 INFO][1][MainProcess][MainThread] - core.schains.process_manager:95 - Fetching schains to monitor... [2023-09-07 16:34:44,995 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:34:44,995 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:34:44,998 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:34:44.998145 [2023-09-07 16:34:45,008 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:34:45,011 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:34:45,017 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:34:45,025 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694104485.0073316, 'te': 1694104485.0251856, 'initial_status': True} ====================================================================================================  [2023-09-07 16:34:45,028 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:34:45.028644 [2023-09-07 16:34:45,036 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:34:45,039 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:34:50,534 INFO][1][MainProcess][MainThread] - core.schains.process_manager:109 - Get leaving_history for node ... [2023-09-07 16:34:52,497 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104488, schain_finish_ts: 1692377616 [2023-09-07 16:34:54,473 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104488, schain_finish_ts: 0 [2023-09-07 16:34:54,854 INFO][1][MainProcess][MainThread] - core.schains.process_manager:117 - Got leaving sChains for the node: [] [2023-09-07 16:34:54,855 INFO][1][MainProcess][MainThread] - core.schains.process_manager:101 -  ==================================================================================================== Monitoring sChains Node ID: 16 sChains on node: [{'name': 'portly-passionate-sirius', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666281384, 'startBlock': 7803380, 'deposit': 0, 'index': 1, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x0000017723ca2', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'curly-red-alterf', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1676058936, 'startBlock': 8469646, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000001fe6cdfa', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'howling-hoedus', 'mainnetOwner': '0x1026467C91fdA40aCc7e6935d1E52f67274FD669', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666181400, 'startBlock': 7796660, 'deposit': 0, 'index': 0, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0xe5a600f4c2608', 'multitransactionMode': False, 'thresholdEncryption': False, 'active': True}, {'name': 'green-giddy-denebola', 'mainnetOwner': '0x8e835d5d9E28ffD259E8Fa8988Cacfc333A582B4', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674565164, 'startBlock': 8368049, 'deposit': 0, 'index': 4, 'generation': 1, 'originator': '0xd2001DAb6898127Be2F167B548691C87251D13C3', 'chainId': '0x00000585eb4b1', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'wan-tasty-kitalpha', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674568392, 'startBlock': 8368271, 'deposit': 0, 'index': 5, 'generation': 1, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x00000152e7be9', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'submissive-well-off-rotanev', 'mainnetOwner': '0x487360568fE5Ee4AE5c75782D5F63e9935B0E512', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1692361104, 'startBlock': 9539053, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000003791fa10', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}] Number of sChains on node: 6 Empty sChain structs: 1 ====================================================================================================  [2023-09-07 16:34:55,232 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:34:55,233 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain portly-passionate-sirius - Process is running: PID = 7 [2023-09-07 16:34:55,601 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:34:55,601 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain curly-red-alterf - Process is running: PID = 8 [2023-09-07 16:34:55,972 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:34:55,972 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain howling-hoedus - Process is running: PID = 9 [2023-09-07 16:34:56,345 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:34:56,345 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain green-giddy-denebola - Process is running: PID = 10 [2023-09-07 16:34:56,714 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:34:56,715 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain wan-tasty-kitalpha - Process is running: PID = 11 [2023-09-07 16:34:57,085 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:34:57,086 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain submissive-well-off-rotanev - Process is running: PID = 24912 [2023-09-07 16:34:57,086 INFO][1][MainProcess][MainThread] - core.schains.process_manager:88 - Process manager procedure finished [2023-09-07 16:34:57,086 INFO][1][MainProcess][MainThread] - __main__:65 - Sleeping for 90s after run_process_manager [2023-09-07 16:34:58,046 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104488, schain_finish_ts: 1692637668 [2023-09-07 16:34:59,212 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:34:59,213 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:34:59,213 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:34:59,214 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:34:59,214 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:34:59,215 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:34:59,218 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:34:59,219 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 16:34:59,248 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:34:59,716 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:34:59,870 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:00,058 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:35:00,063 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2460 [2023-09-07 16:35:00,064 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:35:00,065 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:35:00,065 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:35:00,065 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:35:00,066 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:35:00,071 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:35:00,074 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:35:00.074232 [2023-09-07 16:35:00,244 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:00,277 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:35:00,379 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:35:00,384 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:35:00,392 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:35:00,392 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694104500.2724693, 'te': 1694104500.2864923, 'initial_status': True} ====================================================================================================  [2023-09-07 16:35:00,394 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:35:00.394294 [2023-09-07 16:35:00,398 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:35:00,399 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:35:01,171 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:35:01,171 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:35:01,173 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:35:01.173822 [2023-09-07 16:35:01,185 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:35:01,185 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:35:01,193 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:35:01,199 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694104501.1832473, 'te': 1694104501.1995757, 'initial_status': True} ====================================================================================================  [2023-09-07 16:35:01,202 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:35:01.202068 [2023-09-07 16:35:01,208 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:35:01,208 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:35:01,546 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104500, schain_finish_ts: 1693742004 [2023-09-07 16:35:01,776 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 16:35:01,952 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 16:35:01,952 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 16:35:01,952 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 16:35:01,953 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 16:35:02,130 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 16:35:02,131 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 16:35:02,133 INFO][8][curly-red-alterf][T_1] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 16:35:02,139 ERROR][8][curly-red-alterf][T_1] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 16:35:02,141 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:35:02,695 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:35:02,695 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:35:02,696 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:35:02,696 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:35:02,696 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:35:02,697 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 16:35:02,715 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:35:02,884 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:03,012 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:03,115 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:35:03,116 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2459 [2023-09-07 16:35:03,116 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:35:03,116 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:35:03,117 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:35:03,117 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:35:03,117 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:35:03,123 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:35:03,125 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:35:03.125231 [2023-09-07 16:35:03,257 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:03,282 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:35:03,310 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:35:03,312 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:35:03,319 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:35:03,320 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694104503.280936, 'te': 1694104503.287523, 'initial_status': True} ====================================================================================================  [2023-09-07 16:35:03,322 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:35:03.322021 [2023-09-07 16:35:03,326 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:35:03,327 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:35:06,058 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:40 - Collecting rotation history for submissive-well-off-rotanev... [2023-09-07 16:35:07,203 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:50 - Rotation data for submissive-well-off-rotanev: Rotation(leaving_node_id=65, new_node_id=65, freeze_until=1694108100, rotation_counter=8) [2023-09-07 16:35:07,687 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104500, schain_finish_ts: 1693742304 [2023-09-07 16:35:08,299 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104500, schain_finish_ts: 1693412520 [2023-09-07 16:35:08,793 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:35:08,793 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:35:08,793 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:35:08,793 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 16:35:08,794 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 16:35:08,794 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:35:08,796 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:35:08,798 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 16:35:08,821 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:35:09,101 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:09,320 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:09,410 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:35:09,410 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:35:09,410 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:35:09,410 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:35:09,411 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:35:09,411 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:35:09,412 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:35:09,413 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 16:35:09,439 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:35:09,479 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:35:09,480 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2466 [2023-09-07 16:35:09,481 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:35:09,481 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:35:09,481 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:35:09,481 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:35:09,482 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:35:09,489 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:35:09,491 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:35:09.491159 [2023-09-07 16:35:09,635 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:09,782 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:09,808 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:35:09,843 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:35:09,845 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:35:09,853 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:35:09,853 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694104509.806458, 'te': 1694104509.8124561, 'initial_status': True} ====================================================================================================  [2023-09-07 16:35:09,855 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:35:09.855857 [2023-09-07 16:35:09,862 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:35:09,862 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:35:09,942 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:10,077 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:35:10,078 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2460 [2023-09-07 16:35:10,078 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:35:10,079 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:35:10,079 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:35:10,079 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:35:10,080 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:35:10,086 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:35:10,088 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:35:10.088133 [2023-09-07 16:35:10,267 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:10,295 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:35:10,321 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:35:10,323 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:35:10,330 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:35:10,331 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694104510.2933784, 'te': 1694104510.2997565, 'initial_status': True} ====================================================================================================  [2023-09-07 16:35:10,332 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:35:10.332655 [2023-09-07 16:35:10,336 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:35:10,337 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:35:10,542 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 16:35:10,543 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:35:10,546 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:35:10.546055 [2023-09-07 16:35:10,551 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 16:35:11,161 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:35:11,161 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:35:11,163 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:35:11.163808 [2023-09-07 16:35:11,171 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:35:11,171 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:35:11,176 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:35:11,180 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694104511.1694953, 'te': 1694104511.1807592, 'initial_status': True} ====================================================================================================  [2023-09-07 16:35:11,182 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:35:11.182642 [2023-09-07 16:35:11,187 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:35:11,187 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:35:17,059 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104512, schain_finish_ts: 1692377364 [2023-09-07 16:35:17,405 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104512, schain_finish_ts: 0 [2023-09-07 16:35:18,189 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:35:18,189 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:35:18,189 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:35:18,190 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:35:18,190 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:35:18,190 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:35:18,193 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:35:18,194 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 16:35:18,227 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:35:18,433 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:18,547 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:35:18,548 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:35:18,549 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 16:35:18,706 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:18,872 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:35:18,874 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2444 [2023-09-07 16:35:18,874 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:35:18,874 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:35:18,875 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:35:18,875 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:35:18,875 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:35:18,882 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:35:18,884 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:35:18.884266 [2023-09-07 16:35:19,073 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:19,103 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:35:19,135 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:35:19,137 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:35:19,158 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:35:19,159 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694104519.1002119, 'te': 1694104519.1105092, 'initial_status': True} ====================================================================================================  [2023-09-07 16:35:19,162 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:35:19.162410 [2023-09-07 16:35:19,172 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:35:19,173 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:35:20,089 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:35:20,090 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:35:20,092 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:35:20.092556 [2023-09-07 16:35:20,099 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:35:20,100 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:35:20,107 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:35:20,111 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694104520.0979142, 'te': 1694104520.1115353, 'initial_status': True} ====================================================================================================  [2023-09-07 16:35:20,113 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:35:20.113597 [2023-09-07 16:35:20,124 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:35:20,124 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:35:22,444 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 63 -> 65 [2023-09-07 16:35:27,156 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104524, schain_finish_ts: 1692637668 [2023-09-07 16:35:27,413 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 23 -> 24 [2023-09-07 16:35:28,271 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:35:28,272 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:35:28,272 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:35:28,272 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:35:28,272 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:35:28,273 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:35:28,274 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:35:28,274 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 16:35:28,299 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:35:28,492 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:28,641 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:28,749 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:35:28,750 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2461 [2023-09-07 16:35:28,751 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:35:28,751 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:35:28,751 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:35:28,752 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:35:28,752 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:35:28,760 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:35:28,762 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:35:28.762352 [2023-09-07 16:35:29,056 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:29,101 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:35:29,139 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:35:29,141 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:35:29,150 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:35:29,150 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694104529.0987773, 'te': 1694104529.106114, 'initial_status': True} ====================================================================================================  [2023-09-07 16:35:29,152 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:35:29.152525 [2023-09-07 16:35:29,156 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:35:29,158 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:35:29,429 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 16:35:29,430 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 16:35:29,765 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104524, schain_finish_ts: 1693742304 [2023-09-07 16:35:30,105 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:35:30,105 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:35:30,109 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:35:30.109393 [2023-09-07 16:35:30,120 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:35:30,121 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:35:30,129 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:35:30,135 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694104530.1172295, 'te': 1694104530.1351237, 'initial_status': True} ====================================================================================================  [2023-09-07 16:35:30,142 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:35:30.142683 [2023-09-07 16:35:30,151 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:35:30,152 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:35:30,903 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:35:30,903 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:35:30,903 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:35:30,903 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:35:30,904 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:35:30,904 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 16:35:30,935 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:35:31,398 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:31,634 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:31,779 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:35:31,781 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2467 [2023-09-07 16:35:31,784 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:35:31,785 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:35:31,786 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:35:31,787 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:35:31,788 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:35:31,797 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:35:31,801 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:35:31.801316 [2023-09-07 16:35:32,004 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:32,060 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:35:32,065 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 9 -> 64 [2023-09-07 16:35:32,091 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:35:32,093 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:35:32,100 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:35:32,101 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694104532.0568054, 'te': 1694104532.0655973, 'initial_status': True} ====================================================================================================  [2023-09-07 16:35:32,102 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:35:32.102869 [2023-09-07 16:35:32,107 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:35:32,109 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:35:34,367 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104524, schain_finish_ts: 1693412520 [2023-09-07 16:35:35,554 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:35:35,554 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:35:35,554 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:35:35,554 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:35:35,555 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:35:35,555 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:35:35,558 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:35:35,559 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 16:35:35,583 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:35:36,049 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:36,244 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:36,398 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:35:36,400 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2461 [2023-09-07 16:35:36,401 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:35:36,402 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:35:36,402 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:35:36,403 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:35:36,403 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:35:36,409 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:35:36,411 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:35:36.411754 [2023-09-07 16:35:36,621 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:36,656 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:35:36,705 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:35:36,716 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:35:36,729 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:35:36,729 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694104536.6542723, 'te': 1694104536.6602416, 'initial_status': True} ====================================================================================================  [2023-09-07 16:35:36,731 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:35:36.731334 [2023-09-07 16:35:36,735 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:35:36,736 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:35:37,058 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 35 -> 16 [2023-09-07 16:35:37,405 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:35:37,405 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:35:37,407 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:35:37.407726 [2023-09-07 16:35:37,414 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:35:37,414 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:35:37,420 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:35:37,425 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694104537.4132574, 'te': 1694104537.4247692, 'initial_status': True} ====================================================================================================  [2023-09-07 16:35:37,427 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:35:37.427049 [2023-09-07 16:35:37,433 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:35:37,433 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:35:40,642 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104536, schain_finish_ts: 1693742004 [2023-09-07 16:35:41,759 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:35:41,760 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:35:41,760 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:35:41,760 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 16:35:41,760 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 16:35:41,761 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:35:41,762 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:35:41,763 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 16:35:41,786 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:35:42,092 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:42,228 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:42,325 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 55 -> 63 [2023-09-07 16:35:42,334 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:35:42,335 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2460 [2023-09-07 16:35:42,336 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:35:42,336 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:35:42,337 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:35:42,337 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:35:42,337 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:35:42,343 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:35:42,345 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:35:42.345957 [2023-09-07 16:35:42,538 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:42,564 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:35:42,597 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:35:42,599 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:35:42,606 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:35:42,606 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694104542.56264, 'te': 1694104542.5710583, 'initial_status': True} ====================================================================================================  [2023-09-07 16:35:42,608 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:35:42.608418 [2023-09-07 16:35:42,612 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:35:42,613 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:35:43,590 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 16:35:43,590 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:35:43,593 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:35:43.593850 [2023-09-07 16:35:43,599 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 16:35:44,102 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104536, schain_finish_ts: 1692377364 [2023-09-07 16:35:45,228 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:35:45,228 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:35:45,228 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:35:45,228 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:35:45,229 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:35:45,229 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:35:45,233 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:35:45,234 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 16:35:45,258 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:35:45,476 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:45,650 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:45,806 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:35:45,808 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2445 [2023-09-07 16:35:45,809 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:35:45,809 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:35:45,810 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:35:45,810 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:35:45,811 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:35:45,819 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:35:45,830 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:35:45.830767 [2023-09-07 16:35:46,044 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:35:46,071 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:35:46,101 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:35:46,103 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:35:46,115 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:35:46,116 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694104546.0696883, 'te': 1694104546.0765207, 'initial_status': True} ====================================================================================================  [2023-09-07 16:35:46,117 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:35:46.117884 [2023-09-07 16:35:46,122 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:35:46,123 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:35:47,027 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:35:47,027 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:35:47,030 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:35:47.030325 [2023-09-07 16:35:47,036 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:35:47,036 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:35:47,042 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:35:47,047 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694104547.0351808, 'te': 1694104547.0470815, 'initial_status': True} ====================================================================================================  [2023-09-07 16:35:47,049 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:35:47.049317 [2023-09-07 16:35:47,055 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:35:47,056 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:35:47,979 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 14 -> 11 [2023-09-07 16:35:48,962 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 16:35:49,151 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 16:35:49,152 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 16:35:49,152 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 16:35:49,153 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 16:35:49,330 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 16:35:49,331 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 16:35:49,338 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 16:35:49,348 ERROR][11][wan-tasty-kitalpha][T_0] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 16:35:49,350 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:35:54,122 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 56 -> 60 [2023-09-07 16:35:55,476 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104536, schain_finish_ts: 0 [2023-09-07 16:35:56,617 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:35:56,618 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:35:56,618 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 16:36:00,198 ERROR][24912][submissive-well-off-rotanev][T_0] - core.schains.task:26 - Task submissive-well-off-rotanev-config failed with max() arg is an empty sequence Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 63, in execute self.am.upstream_config() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 194, in upstream_config new_config = create_new_upstream_config( File "/usr/src/admin/core/schains/config/main.py", line 51, in create_new_upstream_config schain_config = generate_schain_config_with_skale( File "/usr/src/admin/core/schains/config/generator.py", line 213, in generate_schain_config_with_skale node_groups = get_previous_schain_groups(skale, schain_name) File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 56, in get_previous_schain_groups _add_previous_schain_rotations_state( File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 119, in _add_previous_schain_rotations_state new_node_id = max(previous_nodes.items(), key=lambda x: x[1]['finish_ts'])[0] ValueError: max() arg is an empty sequence [2023-09-07 16:36:00,199 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:36:02,827 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 16:36:02,828 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 16:36:04,877 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104560, schain_finish_ts: 1693742304 [2023-09-07 16:36:05,196 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104560, schain_finish_ts: 1692637668 [2023-09-07 16:36:06,020 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:36:06,021 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:36:06,021 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:36:06,022 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 16:36:06,022 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 16:36:06,024 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:36:06,029 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:36:06,032 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 16:36:06,078 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:36:06,317 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:36:06,318 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:36:06,318 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:36:06,318 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:36:06,318 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:36:06,319 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:36:06,321 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:36:06,324 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 16:36:06,359 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:36:06,410 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:06,515 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104560, schain_finish_ts: 1693412520 [2023-09-07 16:36:06,565 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:06,779 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:06,893 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:36:06,894 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2468 [2023-09-07 16:36:06,895 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:36:06,895 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:36:06,895 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:36:06,896 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:36:06,899 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:36:06,907 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:36:06,909 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:36:06.909764 [2023-09-07 16:36:06,933 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:07,036 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:36:07,037 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2462 [2023-09-07 16:36:07,038 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:36:07,038 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:36:07,038 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:36:07,039 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:36:07,039 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:36:07,044 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:36:07,046 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:36:07.046935 [2023-09-07 16:36:07,122 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:07,151 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:36:07,184 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:36:07,188 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:36:07,194 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:36:07,195 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694104567.1498652, 'te': 1694104567.1567965, 'initial_status': True} ====================================================================================================  [2023-09-07 16:36:07,196 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:36:07.196868 [2023-09-07 16:36:07,201 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:36:07,201 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:36:07,274 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:07,297 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:36:07,323 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:36:07,325 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:36:07,332 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:36:07,333 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694104567.295467, 'te': 1694104567.3018277, 'initial_status': True} ====================================================================================================  [2023-09-07 16:36:07,334 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:36:07.334572 [2023-09-07 16:36:07,340 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:36:07,341 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:36:07,664 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:36:07,665 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:36:07,665 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:36:07,665 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:36:07,665 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:36:07,665 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:36:07,667 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:36:07,669 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 16:36:07,690 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:36:07,832 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 16:36:07,832 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:36:07,834 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:36:07.834135 [2023-09-07 16:36:07,839 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 16:36:07,884 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:08,062 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:36:08,062 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:36:08,064 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:36:08.064572 [2023-09-07 16:36:08,071 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:08,073 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:36:08,073 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:36:08,084 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:36:08,091 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694104568.0722399, 'te': 1694104568.091028, 'initial_status': True} ====================================================================================================  [2023-09-07 16:36:08,092 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:36:08.092897 [2023-09-07 16:36:08,098 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:36:08,098 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:36:08,197 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:36:08,198 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2462 [2023-09-07 16:36:08,199 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:36:08,199 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:36:08,199 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:36:08,199 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:36:08,200 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:36:08,206 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:36:08,208 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:36:08.208047 [2023-09-07 16:36:08,521 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:08,546 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:36:08,573 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:36:08,575 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:36:08,583 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:36:08,583 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694104568.5444446, 'te': 1694104568.551456, 'initial_status': True} ====================================================================================================  [2023-09-07 16:36:08,585 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:36:08.585496 [2023-09-07 16:36:08,591 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:36:08,591 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:36:09,448 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:36:09,448 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:36:09,450 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:36:09.450885 [2023-09-07 16:36:09,467 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:36:09,467 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:36:09,476 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:36:09,484 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694104569.4650853, 'te': 1694104569.4844546, 'initial_status': True} ====================================================================================================  [2023-09-07 16:36:09,487 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:36:09.487045 [2023-09-07 16:36:09,492 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:36:09,492 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:36:13,672 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104560, schain_finish_ts: 1693742004 [2023-09-07 16:36:14,811 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:36:14,812 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:36:14,812 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:36:14,812 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:36:14,812 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:36:14,813 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 16:36:14,832 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:36:15,090 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:15,304 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:15,434 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:36:15,435 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2461 [2023-09-07 16:36:15,436 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:36:15,436 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:36:15,437 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:36:15,437 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:36:15,437 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:36:15,448 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:36:15,451 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:36:15.451337 [2023-09-07 16:36:15,634 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:15,666 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:36:15,700 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:36:15,703 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:36:15,718 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:36:15,719 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694104575.6646657, 'te': 1694104575.6727824, 'initial_status': True} ====================================================================================================  [2023-09-07 16:36:15,721 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:36:15.721108 [2023-09-07 16:36:15,727 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:36:15,728 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:36:17,190 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104572, schain_finish_ts: 1692377364 [2023-09-07 16:36:18,318 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:36:18,318 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:36:18,318 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:36:18,318 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:36:18,319 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:36:18,319 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:36:18,320 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:36:18,321 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 16:36:18,349 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:36:18,548 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:18,863 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:19,007 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:36:19,009 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2446 [2023-09-07 16:36:19,010 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:36:19,010 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:36:19,010 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:36:19,011 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:36:19,012 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:36:19,019 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:36:19,021 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:36:19.021205 [2023-09-07 16:36:19,224 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:19,253 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:36:19,307 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:36:19,311 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:36:19,338 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:36:19,339 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694104579.2521567, 'te': 1694104579.2599337, 'initial_status': True} ====================================================================================================  [2023-09-07 16:36:19,341 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:36:19.341155 [2023-09-07 16:36:19,346 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:36:19,347 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:36:20,207 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:36:20,207 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:36:20,209 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:36:20.209438 [2023-09-07 16:36:20,218 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:36:20,219 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:36:20,227 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:36:20,232 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694104580.216107, 'te': 1694104580.2317176, 'initial_status': True} ====================================================================================================  [2023-09-07 16:36:20,234 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:36:20.234609 [2023-09-07 16:36:20,239 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:36:20,239 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:36:22,272 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 16:36:22,453 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 16:36:22,454 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 16:36:22,454 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 16:36:22,455 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 16:36:22,628 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 16:36:22,628 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 16:36:22,632 INFO][8][curly-red-alterf][T_1] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 16:36:22,638 ERROR][8][curly-red-alterf][T_1] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 16:36:22,639 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:36:27,135 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 16:36:27,135 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 16:36:27,177 INFO][1][MainProcess][MainThread] - core.schains.cleaner:65 - Cleaner process started [2023-09-07 16:36:27,210 INFO][30409][cleaner][MainThread] - core.schains.cleaner:110 - Cleaner procedure started. [2023-09-07 16:36:29,610 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104584, schain_finish_ts: 1693412520 [2023-09-07 16:36:30,749 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:36:30,749 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:36:30,749 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:36:30,750 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:36:30,751 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:36:30,751 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:36:30,756 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:36:30,757 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 16:36:30,788 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:36:30,985 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:31,183 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:31,293 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:36:31,294 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2463 [2023-09-07 16:36:31,295 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:36:31,295 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:36:31,295 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:36:31,295 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:36:31,296 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:36:31,301 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:36:31,303 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:36:31.303538 [2023-09-07 16:36:31,574 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:31,608 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:36:31,676 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:36:31,681 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:36:31,693 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:36:31,693 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694104591.6066089, 'te': 1694104591.614023, 'initial_status': True} ====================================================================================================  [2023-09-07 16:36:31,696 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:36:31.696732 [2023-09-07 16:36:31,702 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:36:31,703 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:36:32,531 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:36:32,531 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:36:32,533 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:36:32.533727 [2023-09-07 16:36:32,541 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:36:32,541 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:36:32,546 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:36:32,550 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694104592.5397573, 'te': 1694104592.5506194, 'initial_status': True} ====================================================================================================  [2023-09-07 16:36:32,552 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:36:32.552422 [2023-09-07 16:36:32,557 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:36:32,558 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:36:32,862 INFO][30409][cleaner][MainThread] - core.schains.cleaner:116 - sChains on contracts: ['portly-passionate-sirius', 'curly-red-alterf', 'howling-hoedus', 'green-giddy-denebola', 'wan-tasty-kitalpha', '', 'submissive-well-off-rotanev'] sChains on node: ['curly-red-alterf', 'green-giddy-denebola', 'howling-hoedus', 'portly-passionate-sirius', 'submissive-well-off-rotanev', 'wan-tasty-kitalpha'] [2023-09-07 16:36:32,862 INFO][30409][cleaner][MainThread] - core.schains.cleaner:132 - Cleanup procedure finished [2023-09-07 16:36:32,868 INFO][1][MainProcess][MainThread] - core.schains.cleaner:67 - Cleaner process is joined. [2023-09-07 16:36:32,869 INFO][1][MainProcess][MainThread] - core.schains.cleaner:68 - Terminating the process [2023-09-07 16:36:32,869 INFO][1][MainProcess][MainThread] - __main__:70 - Sleeping for 90s after run_cleaner [2023-09-07 16:36:36,776 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104584, schain_finish_ts: 1693742004 [2023-09-07 16:36:37,592 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104596, schain_finish_ts: 0 [2023-09-07 16:36:37,944 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:36:37,945 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:36:37,945 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:36:37,945 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 16:36:37,945 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 16:36:37,945 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:36:37,947 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:36:37,947 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 16:36:37,975 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:36:38,407 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:38,603 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:38,728 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:36:38,729 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2462 [2023-09-07 16:36:38,730 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:36:38,730 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:36:38,730 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:36:38,731 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:36:38,731 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:36:38,737 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:36:38,739 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:36:38.739362 [2023-09-07 16:36:38,761 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:36:38,761 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:36:38,761 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:40 - Task submissive-well-off-rotanev-config finished with None [2023-09-07 16:36:38,762 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:42 - Running task submissive-well-off-rotanev-config [2023-09-07 16:36:38,762 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 16:36:38,892 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:38,932 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:36:38,988 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:36:38,991 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:36:39,001 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:36:39,001 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694104598.9303193, 'te': 1694104598.9371839, 'initial_status': True} ====================================================================================================  [2023-09-07 16:36:39,003 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:36:39.003816 [2023-09-07 16:36:39,008 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:36:39,009 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:36:39,791 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 16:36:39,791 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:36:39,793 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:36:39.793666 [2023-09-07 16:36:39,798 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 16:36:40,021 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104596, schain_finish_ts: 1693742304 [2023-09-07 16:36:40,515 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': False} [2023-09-07 16:36:40,516 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:36:40,518 INFO][24912][submissive-well-off-rotanev][T_0] - web.models.schain:152 - Changing monitor_last_seen for submissive-well-off-rotanev to 2023-09-07 16:36:40.518901 [2023-09-07 16:36:40,525 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.action:190 - Creating new upstream_config rotation_id: 8, stream: regression-2.2 [2023-09-07 16:36:40,527 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.config.main:49 - Generating sChain config for submissive-well-off-rotanev [2023-09-07 16:36:41,156 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:36:41,156 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:36:41,156 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:36:41,157 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:36:41,157 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:36:41,157 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 16:36:41,186 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:36:41,406 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:41,564 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:41,680 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:36:41,681 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2469 [2023-09-07 16:36:41,682 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:36:41,682 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:36:41,683 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:36:41,683 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:36:41,683 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:36:41,689 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:36:41,691 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:36:41.691700 [2023-09-07 16:36:41,916 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:41,950 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:36:41,978 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:36:41,980 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:36:41,988 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:36:41,989 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694104601.949009, 'te': 1694104601.955592, 'initial_status': True} ====================================================================================================  [2023-09-07 16:36:41,990 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:36:41.990489 [2023-09-07 16:36:41,994 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:36:41,995 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:36:44,268 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104596, schain_finish_ts: 1692637668 [2023-09-07 16:36:45,416 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:36:45,416 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:36:45,417 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:36:45,417 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:36:45,417 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:36:45,417 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:36:45,419 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:36:45,419 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 16:36:45,444 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:36:45,632 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:45,910 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:46,017 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:36:46,018 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2463 [2023-09-07 16:36:46,019 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:36:46,019 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:36:46,019 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:36:46,020 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:36:46,020 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:36:46,027 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:36:46,029 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:36:46.029217 [2023-09-07 16:36:46,201 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:36:46,228 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:36:46,258 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:36:46,260 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:36:46,267 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:36:46,268 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694104606.226738, 'te': 1694104606.2331562, 'initial_status': True} ====================================================================================================  [2023-09-07 16:36:46,269 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:36:46.269812 [2023-09-07 16:36:46,274 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:36:46,275 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:36:47,000 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 16:36:47,175 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 16:36:47,175 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 16:36:47,175 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 16:36:47,176 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 16:36:47,193 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:36:47,193 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:36:47,195 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:36:47.195178 [2023-09-07 16:36:47,203 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:36:47,203 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:36:47,209 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:36:47,214 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694104607.2015848, 'te': 1694104607.2142055, 'initial_status': True} ====================================================================================================  [2023-09-07 16:36:47,216 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:36:47.216032 [2023-09-07 16:36:47,220 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:36:47,220 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:36:47,345 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 16:36:47,346 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 16:36:47,348 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 16:36:47,352 ERROR][11][wan-tasty-kitalpha][T_1] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 16:36:47,353 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:36:58,546 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 16:36:58,547 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 16:36:59,259 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104608, schain_finish_ts: 1692377364 [2023-09-07 16:37:00,413 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:37:00,414 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:37:00,414 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:37:00,414 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:37:00,414 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:37:00,415 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:37:00,416 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:37:00,417 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 16:37:00,439 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:37:00,623 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:00,700 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104608, schain_finish_ts: 1693412520 [2023-09-07 16:37:00,773 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:00,877 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:37:00,878 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2447 [2023-09-07 16:37:00,878 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:37:00,878 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:37:00,879 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:37:00,879 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:37:00,880 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:37:00,886 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:37:00,888 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:37:00.888288 [2023-09-07 16:37:01,037 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:01,065 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:37:01,105 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:37:01,107 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:37:01,121 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:37:01,121 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694104621.0629237, 'te': 1694104621.0710342, 'initial_status': True} ====================================================================================================  [2023-09-07 16:37:01,124 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:37:01.124280 [2023-09-07 16:37:01,130 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:37:01,131 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:37:01,816 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:37:01,817 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:37:01,817 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:37:01,818 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:37:01,818 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:37:01,819 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:37:01,821 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:37:01,824 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 16:37:01,868 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:37:02,136 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:02,198 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:37:02,202 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:37:02,205 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:37:02.205273 [2023-09-07 16:37:02,215 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:37:02,216 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:37:02,222 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:37:02,228 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694104622.214446, 'te': 1694104622.228005, 'initial_status': True} ====================================================================================================  [2023-09-07 16:37:02,234 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:37:02.234501 [2023-09-07 16:37:02,240 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:37:02,241 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:37:02,442 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:02,623 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:37:02,624 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2464 [2023-09-07 16:37:02,625 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:37:02,625 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:37:02,625 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:37:02,625 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:37:02,626 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:37:02,634 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:37:02,636 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:37:02.636630 [2023-09-07 16:37:02,745 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104620, schain_finish_ts: 0 [2023-09-07 16:37:02,848 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:02,874 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:37:02,917 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:37:02,921 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:37:02,932 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:37:02,933 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694104622.8723495, 'te': 1694104622.8834062, 'initial_status': True} ====================================================================================================  [2023-09-07 16:37:02,935 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:37:02.935775 [2023-09-07 16:37:02,947 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:37:02,948 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:37:03,679 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:37:03,679 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:37:03,682 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:37:03.682618 [2023-09-07 16:37:03,690 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:37:03,690 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:37:03,698 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:37:03,703 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694104623.6885114, 'te': 1694104623.7034357, 'initial_status': True} ====================================================================================================  [2023-09-07 16:37:03,705 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:37:03.705460 [2023-09-07 16:37:03,710 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:37:03,710 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:37:03,877 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:37:03,877 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:37:03,877 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 16:37:04,966 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104620, schain_finish_ts: 1693742004 [2023-09-07 16:37:06,089 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:37:06,089 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:37:06,089 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:37:06,090 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:37:06,090 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:37:06,090 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 16:37:06,113 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:37:06,302 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:06,592 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:06,782 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:37:06,787 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2463 [2023-09-07 16:37:06,789 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:37:06,791 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:37:06,792 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:37:06,794 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:37:06,796 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:37:06,818 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:37:06,844 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:37:06.844785 [2023-09-07 16:37:07,177 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:07,204 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:37:07,235 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:37:07,241 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:37:07,249 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:37:07,249 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694104627.2029307, 'te': 1694104627.2099793, 'initial_status': True} ====================================================================================================  [2023-09-07 16:37:07,251 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:37:07.251165 [2023-09-07 16:37:07,255 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:37:07,255 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:37:17,935 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 16:37:18,125 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 16:37:18,125 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 16:37:18,125 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 16:37:18,126 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 16:37:18,301 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 16:37:18,301 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 16:37:18,303 INFO][8][curly-red-alterf][T_0] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 16:37:18,307 ERROR][8][curly-red-alterf][T_0] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 16:37:18,308 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:37:19,355 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104632, schain_finish_ts: 1692637668 [2023-09-07 16:37:20,499 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:37:20,499 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:37:20,499 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:37:20,500 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:37:20,500 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:37:20,500 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:37:20,502 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:37:20,504 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 16:37:20,528 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:37:20,781 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:20,943 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:21,056 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:37:21,057 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2464 [2023-09-07 16:37:21,058 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:37:21,058 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:37:21,058 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:37:21,058 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:37:21,059 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:37:21,064 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:37:21,066 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:37:21.066514 [2023-09-07 16:37:21,231 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:21,259 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:37:21,287 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:37:21,288 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:37:21,297 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:37:21,298 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694104641.2575388, 'te': 1694104641.2632504, 'initial_status': True} ====================================================================================================  [2023-09-07 16:37:21,300 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:37:21.300592 [2023-09-07 16:37:21,306 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:37:21,307 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:37:22,111 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104632, schain_finish_ts: 1693742304 [2023-09-07 16:37:22,312 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:37:22,313 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:37:22,315 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:37:22.315446 [2023-09-07 16:37:22,323 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:37:22,323 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:37:22,330 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:37:22,335 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694104642.3217943, 'te': 1694104642.3350897, 'initial_status': True} ====================================================================================================  [2023-09-07 16:37:22,336 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:37:22.336827 [2023-09-07 16:37:22,341 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:37:22,341 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:37:23,231 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:37:23,231 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:37:23,231 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:37:23,231 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 16:37:23,232 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 16:37:23,232 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:37:23,234 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:37:23,239 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 16:37:23,261 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:37:23,469 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:23,656 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:23,768 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:37:23,769 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2470 [2023-09-07 16:37:23,769 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:37:23,770 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:37:23,770 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:37:23,770 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:37:23,771 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:37:23,776 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:37:23,778 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:37:23.778769 [2023-09-07 16:37:24,061 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:24,088 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:37:24,125 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:37:24,126 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:37:24,134 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:37:24,134 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694104644.087073, 'te': 1694104644.0945275, 'initial_status': True} ====================================================================================================  [2023-09-07 16:37:24,136 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:37:24.136011 [2023-09-07 16:37:24,141 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:37:24,142 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:37:25,079 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 16:37:25,080 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:37:25,082 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:37:25.082442 [2023-09-07 16:37:25,088 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 16:37:29,355 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104644, schain_finish_ts: 1692377364 [2023-09-07 16:37:30,489 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:37:30,489 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:37:30,489 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:37:30,489 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:37:30,489 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:37:30,490 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:37:30,493 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:37:30,496 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 16:37:30,516 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:37:30,873 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:31,055 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:31,176 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:37:31,177 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2448 [2023-09-07 16:37:31,178 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:37:31,178 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:37:31,178 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:37:31,178 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:37:31,179 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:37:31,185 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:37:31,189 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:37:31.189025 [2023-09-07 16:37:31,362 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:31,443 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:37:31,486 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:37:31,488 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:37:31,497 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:37:31,497 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694104651.4422257, 'te': 1694104651.449922, 'initial_status': True} ====================================================================================================  [2023-09-07 16:37:31,499 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:37:31.499467 [2023-09-07 16:37:31,504 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:37:31,505 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:37:32,360 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:37:32,360 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:37:32,362 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:37:32.362489 [2023-09-07 16:37:32,369 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:37:32,369 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:37:32,376 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:37:32,381 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694104652.367361, 'te': 1694104652.381078, 'initial_status': True} ====================================================================================================  [2023-09-07 16:37:32,382 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:37:32.382939 [2023-09-07 16:37:32,388 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:37:32,388 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:37:34,778 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104644, schain_finish_ts: 1693412520 [2023-09-07 16:37:35,908 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:37:35,908 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:37:35,908 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:37:35,908 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:37:35,909 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:37:35,909 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:37:35,911 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:37:35,911 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 16:37:35,939 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:37:36,143 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:36,331 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:36,449 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:37:36,451 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2465 [2023-09-07 16:37:36,451 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:37:36,452 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:37:36,452 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:37:36,452 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:37:36,453 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:37:36,458 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:37:36,460 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:37:36.460689 [2023-09-07 16:37:36,609 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:36,644 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:37:36,673 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:37:36,675 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:37:36,682 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:37:36,683 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694104656.6424766, 'te': 1694104656.6500177, 'initial_status': True} ====================================================================================================  [2023-09-07 16:37:36,684 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:37:36.684768 [2023-09-07 16:37:36,692 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:37:36,693 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:37:37,711 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:37:37,712 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:37:37,714 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:37:37.714238 [2023-09-07 16:37:37,721 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:37:37,721 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:37:37,727 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:37:37,732 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694104657.720066, 'te': 1694104657.7321613, 'initial_status': True} ====================================================================================================  [2023-09-07 16:37:37,735 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:37:37.735108 [2023-09-07 16:37:37,740 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:37:37,740 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:37:37,847 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104656, schain_finish_ts: 0 [2023-09-07 16:37:38,981 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:37:38,981 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:37:38,982 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 16:37:44,169 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 16:37:44,170 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 16:37:45,046 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104656, schain_finish_ts: 1693742004 [2023-09-07 16:37:46,175 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:37:46,175 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:37:46,176 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:37:46,176 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 16:37:46,176 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 16:37:46,176 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:37:46,179 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:37:46,181 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 16:37:46,202 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:37:46,383 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:46,408 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104656, schain_finish_ts: 1692637668 [2023-09-07 16:37:46,526 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:46,630 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:37:46,631 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2464 [2023-09-07 16:37:46,631 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:37:46,632 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:37:46,632 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:37:46,632 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:37:46,633 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:37:46,639 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:37:46,641 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:37:46.641526 [2023-09-07 16:37:46,811 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:46,837 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:37:46,868 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:37:46,870 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:37:46,879 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:37:46,879 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694104666.8337345, 'te': 1694104666.8464553, 'initial_status': True} ====================================================================================================  [2023-09-07 16:37:46,881 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:37:46.881243 [2023-09-07 16:37:46,886 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:37:46,886 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:37:47,508 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:37:47,508 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:37:47,508 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:37:47,509 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:37:47,509 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:37:47,509 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:37:47,510 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:37:47,511 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 16:37:47,533 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:37:47,710 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:47,889 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 16:37:47,890 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:37:47,892 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:37:47.892010 [2023-09-07 16:37:47,898 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 16:37:47,977 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:48,076 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:37:48,077 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2465 [2023-09-07 16:37:48,078 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:37:48,079 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:37:48,079 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:37:48,079 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:37:48,080 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:37:48,086 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:37:48,088 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:37:48.088571 [2023-09-07 16:37:48,246 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:48,271 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:37:48,301 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:37:48,304 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:37:48,311 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:37:48,311 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694104668.2698045, 'te': 1694104668.276471, 'initial_status': True} ====================================================================================================  [2023-09-07 16:37:48,313 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:37:48.313556 [2023-09-07 16:37:48,317 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:37:48,318 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:37:49,161 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104656, schain_finish_ts: 1693742304 [2023-09-07 16:37:49,297 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:37:49,297 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:37:49,306 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:37:49.306135 [2023-09-07 16:37:49,316 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:37:49,316 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:37:49,323 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:37:49,328 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694104669.312815, 'te': 1694104669.3287332, 'initial_status': True} ====================================================================================================  [2023-09-07 16:37:49,330 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:37:49.330579 [2023-09-07 16:37:49,335 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:37:49,336 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:37:50,286 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:37:50,286 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:37:50,287 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:37:50,287 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:37:50,287 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:37:50,287 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 16:37:50,337 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:37:50,587 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:50,818 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:50,933 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:37:50,934 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2471 [2023-09-07 16:37:50,935 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:37:50,935 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:37:50,935 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:37:50,935 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:37:50,936 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:37:50,942 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:37:50,945 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:37:50.945184 [2023-09-07 16:37:51,122 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:51,151 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:37:51,187 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:37:51,189 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:37:51,202 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:37:51,204 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694104671.1495774, 'te': 1694104671.156807, 'initial_status': True} ====================================================================================================  [2023-09-07 16:37:51,207 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:37:51.207917 [2023-09-07 16:37:51,220 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:37:51,221 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:37:51,460 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104668, schain_finish_ts: 1692377364 [2023-09-07 16:37:52,610 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:37:52,611 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:37:52,611 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:37:52,612 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:37:52,612 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:37:52,613 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:37:52,615 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:37:52,620 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 16:37:52,655 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:37:52,939 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:53,160 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:53,352 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:37:53,354 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2449 [2023-09-07 16:37:53,354 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:37:53,355 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:37:53,355 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:37:53,356 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:37:53,357 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:37:53,369 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:37:53,372 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:37:53.372724 [2023-09-07 16:37:53,690 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:37:53,776 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:37:53,824 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:37:53,826 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:37:53,837 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:37:53,838 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694104673.77357, 'te': 1694104673.792847, 'initial_status': True} ====================================================================================================  [2023-09-07 16:37:53,841 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:37:53.841218 [2023-09-07 16:37:53,848 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:37:53,849 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:37:54,453 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:37:54,453 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:37:54,457 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:37:54.457009 [2023-09-07 16:37:54,466 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:37:54,466 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:37:54,481 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:37:54,490 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694104674.464135, 'te': 1694104674.490189, 'initial_status': True} ====================================================================================================  [2023-09-07 16:37:54,492 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:37:54.492800 [2023-09-07 16:37:54,499 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:37:54,500 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:38:02,948 INFO][1][MainProcess][MainThread] - core.schains.process_manager:57 - Process manager started [2023-09-07 16:38:03,165 INFO][1][MainProcess][MainThread] - core.schains.notifications:35 - Node account has 406869999689179237 WEI [2023-09-07 16:38:03,166 INFO][1][MainProcess][MainThread] - tools.notifications.messages:188 - Saving new balance state 3004 1 [2023-09-07 16:38:03,166 INFO][1][MainProcess][MainThread] - core.schains.process_manager:95 - Fetching schains to monitor... [2023-09-07 16:38:03,293 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 16:38:03,467 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 16:38:03,468 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 16:38:03,468 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 16:38:03,468 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 16:38:03,648 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 16:38:03,649 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 16:38:03,655 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 16:38:03,661 ERROR][11][wan-tasty-kitalpha][T_0] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 16:38:03,662 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:38:06,675 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 16:38:06,675 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 16:38:07,156 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104668, schain_finish_ts: 1693742004 [2023-09-07 16:38:08,269 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:38:08,270 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:38:08,270 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:38:08,270 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:38:08,271 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:38:08,271 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 16:38:08,290 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:38:08,505 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:08,645 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:08,752 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:38:08,753 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2465 [2023-09-07 16:38:08,753 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:38:08,754 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:38:08,754 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:38:08,754 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:38:08,755 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:38:08,761 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:38:08,769 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:38:08.769871 [2023-09-07 16:38:08,869 INFO][1][MainProcess][MainThread] - core.schains.process_manager:109 - Get leaving_history for node ... [2023-09-07 16:38:09,054 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:09,107 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:38:09,142 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:38:09,144 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:38:09,152 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:38:09,152 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694104689.1053584, 'te': 1694104689.114212, 'initial_status': True} ====================================================================================================  [2023-09-07 16:38:09,154 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:38:09.154457 [2023-09-07 16:38:09,159 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:38:09,160 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:38:09,858 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104668, schain_finish_ts: 1693412520 [2023-09-07 16:38:09,912 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104668, schain_finish_ts: 0 [2023-09-07 16:38:10,827 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104668, schain_finish_ts: 1692377616 [2023-09-07 16:38:10,986 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:38:10,987 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:38:10,987 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:38:10,988 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:38:10,988 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:38:10,988 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:38:10,990 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:38:10,994 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 16:38:11,018 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:38:11,018 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:38:11,019 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 16:38:11,020 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:38:11,326 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:11,471 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:11,587 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:38:11,588 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2466 [2023-09-07 16:38:11,589 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:38:11,589 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:38:11,589 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:38:11,590 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:38:11,590 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:38:11,595 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:38:11,597 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:38:11.597966 [2023-09-07 16:38:11,751 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:11,782 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:38:11,817 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:38:11,819 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:38:11,827 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:38:11,827 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694104691.7799954, 'te': 1694104691.7912536, 'initial_status': True} ====================================================================================================  [2023-09-07 16:38:11,829 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:38:11.829412 [2023-09-07 16:38:11,835 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:38:11,838 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:38:12,741 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:38:12,741 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:38:12,743 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:38:12.743699 [2023-09-07 16:38:12,754 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:38:12,754 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:38:12,763 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:38:12,769 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694104692.7532096, 'te': 1694104692.7689452, 'initial_status': True} ====================================================================================================  [2023-09-07 16:38:12,770 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:38:12.770847 [2023-09-07 16:38:12,780 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:38:12,781 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:38:12,790 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104668, schain_finish_ts: 0 [2023-09-07 16:38:13,165 INFO][1][MainProcess][MainThread] - core.schains.process_manager:117 - Got leaving sChains for the node: [] [2023-09-07 16:38:13,166 INFO][1][MainProcess][MainThread] - core.schains.process_manager:101 -  ==================================================================================================== Monitoring sChains Node ID: 16 sChains on node: [{'name': 'portly-passionate-sirius', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666281384, 'startBlock': 7803380, 'deposit': 0, 'index': 1, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x0000017723ca2', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'curly-red-alterf', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1676058936, 'startBlock': 8469646, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000001fe6cdfa', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'howling-hoedus', 'mainnetOwner': '0x1026467C91fdA40aCc7e6935d1E52f67274FD669', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666181400, 'startBlock': 7796660, 'deposit': 0, 'index': 0, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0xe5a600f4c2608', 'multitransactionMode': False, 'thresholdEncryption': False, 'active': True}, {'name': 'green-giddy-denebola', 'mainnetOwner': '0x8e835d5d9E28ffD259E8Fa8988Cacfc333A582B4', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674565164, 'startBlock': 8368049, 'deposit': 0, 'index': 4, 'generation': 1, 'originator': '0xd2001DAb6898127Be2F167B548691C87251D13C3', 'chainId': '0x00000585eb4b1', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'wan-tasty-kitalpha', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674568392, 'startBlock': 8368271, 'deposit': 0, 'index': 5, 'generation': 1, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x00000152e7be9', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'submissive-well-off-rotanev', 'mainnetOwner': '0x487360568fE5Ee4AE5c75782D5F63e9935B0E512', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1692361104, 'startBlock': 9539053, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000003791fa10', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}] Number of sChains on node: 6 Empty sChain structs: 1 ====================================================================================================  [2023-09-07 16:38:13,541 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:38:13,542 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain portly-passionate-sirius - Process is running: PID = 7 [2023-09-07 16:38:13,910 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:38:13,910 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain curly-red-alterf - Process is running: PID = 8 [2023-09-07 16:38:14,287 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:38:14,288 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain howling-hoedus - Process is running: PID = 9 [2023-09-07 16:38:14,660 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:38:14,660 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain green-giddy-denebola - Process is running: PID = 10 [2023-09-07 16:38:15,047 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:38:15,048 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain wan-tasty-kitalpha - Process is running: PID = 11 [2023-09-07 16:38:15,422 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:38:15,423 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain submissive-well-off-rotanev - Process is running: PID = 24912 [2023-09-07 16:38:15,423 INFO][1][MainProcess][MainThread] - core.schains.process_manager:88 - Process manager procedure finished [2023-09-07 16:38:15,423 INFO][1][MainProcess][MainThread] - __main__:65 - Sleeping for 90s after run_process_manager [2023-09-07 16:38:22,451 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104668, schain_finish_ts: 1692637668 [2023-09-07 16:38:23,582 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:38:23,582 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:38:23,582 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:38:23,583 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:38:23,583 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:38:23,583 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:38:23,585 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:38:23,585 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 16:38:23,610 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:38:23,816 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:23,980 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:24,081 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:38:24,082 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2466 [2023-09-07 16:38:24,083 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:38:24,083 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:38:24,083 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:38:24,083 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:38:24,084 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:38:24,090 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:38:24,091 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:38:24.091966 [2023-09-07 16:38:24,249 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:24,273 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:38:24,309 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:38:24,311 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:38:24,318 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:38:24,318 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694104704.2721612, 'te': 1694104704.278339, 'initial_status': True} ====================================================================================================  [2023-09-07 16:38:24,320 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:38:24.320468 [2023-09-07 16:38:24,324 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:38:24,325 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:38:25,367 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:38:25,368 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:38:25,370 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:38:25.370712 [2023-09-07 16:38:25,380 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:38:25,380 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:38:25,388 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:38:25,392 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694104705.379287, 'te': 1694104705.392399, 'initial_status': True} ====================================================================================================  [2023-09-07 16:38:25,394 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:38:25.394448 [2023-09-07 16:38:25,399 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:38:25,399 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:38:26,380 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 16:38:26,568 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 16:38:26,569 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 16:38:26,569 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 16:38:26,569 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 16:38:26,735 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 16:38:26,735 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 16:38:26,737 INFO][8][curly-red-alterf][T_1] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 16:38:26,742 ERROR][8][curly-red-alterf][T_1] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 16:38:26,743 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:38:28,203 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104704, schain_finish_ts: 1693742304 [2023-09-07 16:38:28,587 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104704, schain_finish_ts: 1692377364 [2023-09-07 16:38:29,319 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:38:29,319 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:38:29,319 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:38:29,319 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 16:38:29,320 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 16:38:29,320 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:38:29,321 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:38:29,324 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 16:38:29,345 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:38:29,580 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:29,737 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:38:29,738 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:38:29,738 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:38:29,738 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:38:29,738 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:38:29,739 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:38:29,740 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:38:29,742 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 16:38:29,749 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:29,769 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:38:29,857 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:38:29,858 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2472 [2023-09-07 16:38:29,859 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:38:29,859 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:38:29,859 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:38:29,860 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:38:29,860 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:38:29,866 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:38:29,868 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:38:29.868629 [2023-09-07 16:38:30,169 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:30,489 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:30,521 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:38:30,545 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:40 - Collecting rotation history for submissive-well-off-rotanev... [2023-09-07 16:38:30,561 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:38:30,563 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:38:30,571 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:38:30,572 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694104710.5198033, 'te': 1694104710.5325418, 'initial_status': True} ====================================================================================================  [2023-09-07 16:38:30,575 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:38:30.575322 [2023-09-07 16:38:30,588 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:38:30,589 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:38:30,687 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:30,834 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:38:30,835 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2450 [2023-09-07 16:38:30,836 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:38:30,836 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:38:30,836 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:38:30,837 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:38:30,837 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:38:30,858 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:38:30,861 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:38:30.861329 [2023-09-07 16:38:31,097 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:31,134 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:38:31,143 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 16:38:31,143 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:38:31,145 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:38:31.145703 [2023-09-07 16:38:31,150 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 16:38:31,184 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:38:31,188 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:38:31,195 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:38:31,196 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694104711.1322715, 'te': 1694104711.1400537, 'initial_status': True} ====================================================================================================  [2023-09-07 16:38:31,197 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:38:31.197649 [2023-09-07 16:38:31,202 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:38:31,202 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:38:31,618 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:38:31,618 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:38:31,620 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:38:31.620161 [2023-09-07 16:38:31,629 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:38:31,629 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:38:31,636 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:38:31,641 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694104711.6277323, 'te': 1694104711.6410732, 'initial_status': True} ====================================================================================================  [2023-09-07 16:38:31,643 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:38:31.643942 [2023-09-07 16:38:31,649 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:38:31,649 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:38:31,725 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:50 - Rotation data for submissive-well-off-rotanev: Rotation(leaving_node_id=65, new_node_id=65, freeze_until=1694108100, rotation_counter=8) [2023-09-07 16:38:36,989 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104704, schain_finish_ts: 0 [2023-09-07 16:38:38,196 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:38:38,202 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:38:38,204 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 16:38:39,992 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104716, schain_finish_ts: 1693412520 [2023-09-07 16:38:41,146 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:38:41,147 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:38:41,147 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:38:41,147 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:38:41,148 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:38:41,148 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:38:41,151 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:38:41,153 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 16:38:41,191 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:38:41,372 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:41,592 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:41,735 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:38:41,736 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2467 [2023-09-07 16:38:41,737 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:38:41,737 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:38:41,738 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:38:41,738 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:38:41,738 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:38:41,744 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:38:41,746 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:38:41.746491 [2023-09-07 16:38:41,916 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:41,974 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:38:42,023 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:38:42,026 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:38:42,035 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:38:42,036 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694104721.9695861, 'te': 1694104721.9845977, 'initial_status': True} ====================================================================================================  [2023-09-07 16:38:42,038 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:38:42.038706 [2023-09-07 16:38:42,045 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:38:42,045 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:38:42,948 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:38:42,948 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:38:42,951 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:38:42.951079 [2023-09-07 16:38:42,961 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:38:42,961 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:38:42,969 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:38:42,975 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694104722.9590967, 'te': 1694104722.9756386, 'initial_status': True} ====================================================================================================  [2023-09-07 16:38:42,979 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:38:42.979249 [2023-09-07 16:38:42,984 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:38:42,984 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:38:46,983 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 63 -> 65 [2023-09-07 16:38:49,213 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104716, schain_finish_ts: 1693742004 [2023-09-07 16:38:50,253 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 16:38:50,253 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 16:38:50,343 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:38:50,344 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:38:50,344 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:38:50,344 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 16:38:50,344 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 16:38:50,345 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:38:50,346 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:38:50,347 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 16:38:50,371 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:38:50,559 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:50,709 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:50,829 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:38:50,830 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2466 [2023-09-07 16:38:50,831 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:38:50,831 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:38:50,831 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:38:50,831 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:38:50,832 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:38:50,837 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:38:50,839 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:38:50.839852 [2023-09-07 16:38:51,014 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:51,046 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:38:51,088 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:38:51,090 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:38:51,097 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:38:51,098 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694104731.0443656, 'te': 1694104731.0601566, 'initial_status': True} ====================================================================================================  [2023-09-07 16:38:51,099 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:38:51.099818 [2023-09-07 16:38:51,104 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:38:51,105 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:38:51,881 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 23 -> 24 [2023-09-07 16:38:52,147 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 16:38:52,147 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:38:52,149 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:38:52.149894 [2023-09-07 16:38:52,155 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 16:38:54,719 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104716, schain_finish_ts: 1692377364 [2023-09-07 16:38:55,870 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:38:55,870 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:38:55,870 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:38:55,870 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:38:55,870 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:38:55,871 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:38:55,873 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:38:55,876 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 16:38:55,908 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:38:56,119 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:56,309 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:56,433 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:38:56,435 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2451 [2023-09-07 16:38:56,438 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:38:56,441 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:38:56,444 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:38:56,446 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:38:56,447 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:38:56,458 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:38:56,460 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:38:56.460916 [2023-09-07 16:38:56,478 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 9 -> 64 [2023-09-07 16:38:56,761 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:56,800 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:38:56,847 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:38:56,849 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:38:56,856 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:38:56,857 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694104736.7992175, 'te': 1694104736.8120675, 'initial_status': True} ====================================================================================================  [2023-09-07 16:38:56,858 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:38:56.858591 [2023-09-07 16:38:56,864 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:38:56,865 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:38:57,632 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:38:57,632 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:38:57,634 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:38:57.634501 [2023-09-07 16:38:57,642 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:38:57,642 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:38:57,649 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:38:57,654 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694104737.6409988, 'te': 1694104737.6541924, 'initial_status': True} ====================================================================================================  [2023-09-07 16:38:57,656 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:38:57.655981 [2023-09-07 16:38:57,661 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:38:57,661 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:38:58,241 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104716, schain_finish_ts: 1693742304 [2023-09-07 16:38:59,351 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:38:59,352 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:38:59,352 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:38:59,352 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:38:59,352 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:38:59,352 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 16:38:59,372 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:38:59,602 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:59,755 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:38:59,863 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:38:59,864 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2473 [2023-09-07 16:38:59,864 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:38:59,864 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:38:59,865 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:38:59,865 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:38:59,865 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:38:59,871 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:38:59,873 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:38:59.873485 [2023-09-07 16:39:00,047 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:39:00,072 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:39:00,103 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:39:00,105 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:39:00,115 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:39:00,115 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694104740.0707374, 'te': 1694104740.0797517, 'initial_status': True} ====================================================================================================  [2023-09-07 16:39:00,117 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:39:00.117175 [2023-09-07 16:39:00,124 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:39:00,124 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:39:01,346 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 35 -> 16 [2023-09-07 16:39:03,532 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104740, schain_finish_ts: 1692637668 [2023-09-07 16:39:04,659 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:39:04,660 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:39:04,660 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:39:04,660 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:39:04,661 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:39:04,661 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:39:04,664 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:39:04,667 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 16:39:04,709 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:39:05,176 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:39:05,438 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:39:05,600 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:39:05,606 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2467 [2023-09-07 16:39:05,607 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:39:05,607 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:39:05,608 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:39:05,608 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:39:05,609 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:39:05,615 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:39:05,618 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:39:05.618003 [2023-09-07 16:39:05,783 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:39:05,810 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:39:05,842 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:39:05,844 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:39:05,852 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:39:05,854 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694104745.8091688, 'te': 1694104745.8173494, 'initial_status': True} ====================================================================================================  [2023-09-07 16:39:05,858 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:39:05.857998 [2023-09-07 16:39:05,867 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:39:05,868 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:39:06,588 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:39:06,589 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:39:06,591 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:39:06.591974 [2023-09-07 16:39:06,602 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:39:06,602 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:39:06,610 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:39:06,617 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694104746.599425, 'te': 1694104746.6166675, 'initial_status': True} ====================================================================================================  [2023-09-07 16:39:06,620 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:39:06.620935 [2023-09-07 16:39:06,626 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 55 -> 63 [2023-09-07 16:39:06,627 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:39:06,628 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:39:09,082 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104740, schain_finish_ts: 1693412520 [2023-09-07 16:39:09,530 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 16:39:09,708 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 16:39:09,711 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 16:39:09,711 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 16:39:09,712 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 16:39:09,901 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 16:39:09,902 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 16:39:09,905 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 16:39:09,910 ERROR][11][wan-tasty-kitalpha][T_1] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 16:39:09,911 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:39:10,210 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:39:10,210 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:39:10,210 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:39:10,211 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:39:10,211 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:39:10,211 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:39:10,212 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:39:10,215 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 16:39:10,258 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:39:10,656 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:39:10,815 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:39:10,947 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:39:10,952 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2468 [2023-09-07 16:39:10,953 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:39:10,953 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:39:10,953 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:39:10,954 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:39:10,954 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:39:10,959 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:39:10,961 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:39:10.961872 [2023-09-07 16:39:11,104 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:39:11,130 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:39:11,142 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 16:39:11,142 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 16:39:11,158 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:39:11,160 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:39:11,170 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:39:11,171 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694104751.1289172, 'te': 1694104751.13541, 'initial_status': True} ====================================================================================================  [2023-09-07 16:39:11,173 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:39:11.173246 [2023-09-07 16:39:11,178 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:39:11,179 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:39:12,017 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:39:12,017 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:39:12,019 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:39:12.019583 [2023-09-07 16:39:12,026 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:39:12,027 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:39:12,032 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:39:12,037 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694104752.0253553, 'te': 1694104752.0372367, 'initial_status': True} ====================================================================================================  [2023-09-07 16:39:12,039 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:39:12.039135 [2023-09-07 16:39:12,043 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:39:12,043 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:39:12,293 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 14 -> 11 [2023-09-07 16:39:14,172 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104740, schain_finish_ts: 0 [2023-09-07 16:39:15,281 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:39:15,281 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:39:15,281 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 16:39:18,557 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 56 -> 60 [2023-09-07 16:39:22,286 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104740, schain_finish_ts: 1693742304 [2023-09-07 16:39:23,424 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:39:23,424 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:39:23,424 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:39:23,425 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 16:39:23,425 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 16:39:23,429 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:39:23,433 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:39:23,435 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 16:39:23,473 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:39:23,775 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:39:24,122 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:39:24,243 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:39:24,244 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2474 [2023-09-07 16:39:24,245 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:39:24,245 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:39:24,246 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:39:24,246 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:39:24,246 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:39:24,252 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:39:24,254 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:39:24.254600 [2023-09-07 16:39:24,394 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:39:24,419 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:39:24,456 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:39:24,458 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:39:24,465 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:39:24,466 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694104764.4176414, 'te': 1694104764.4275424, 'initial_status': True} ====================================================================================================  [2023-09-07 16:39:24,467 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:39:24.467731 [2023-09-07 16:39:24,475 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:39:24,481 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:39:24,725 ERROR][24912][submissive-well-off-rotanev][T_0] - core.schains.task:26 - Task submissive-well-off-rotanev-config failed with max() arg is an empty sequence Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 63, in execute self.am.upstream_config() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 194, in upstream_config new_config = create_new_upstream_config( File "/usr/src/admin/core/schains/config/main.py", line 51, in create_new_upstream_config schain_config = generate_schain_config_with_skale( File "/usr/src/admin/core/schains/config/generator.py", line 213, in generate_schain_config_with_skale node_groups = get_previous_schain_groups(skale, schain_name) File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 56, in get_previous_schain_groups _add_previous_schain_rotations_state( File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 119, in _add_previous_schain_rotations_state new_node_id = max(previous_nodes.items(), key=lambda x: x[1]['finish_ts'])[0] ValueError: max() arg is an empty sequence [2023-09-07 16:39:24,725 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:39:25,281 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104740, schain_finish_ts: 1693742004 [2023-09-07 16:39:25,287 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 16:39:25,287 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:39:25,291 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:39:25.291106 [2023-09-07 16:39:25,296 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 16:39:26,416 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:39:26,417 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:39:26,417 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:39:26,417 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:39:26,417 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:39:26,417 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 16:39:26,450 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:39:26,712 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:39:27,116 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:39:27,220 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:39:27,221 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2467 [2023-09-07 16:39:27,223 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:39:27,225 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:39:27,225 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:39:27,226 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:39:27,228 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:39:27,235 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:39:27,237 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:39:27.237111 [2023-09-07 16:39:27,412 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:39:27,454 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:39:27,481 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:39:27,484 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:39:27,491 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:39:27,491 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694104767.4524493, 'te': 1694104767.4592373, 'initial_status': True} ====================================================================================================  [2023-09-07 16:39:27,493 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:39:27.493076 [2023-09-07 16:39:27,497 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:39:27,498 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:39:30,603 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 16:39:30,610 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104764, schain_finish_ts: 1692637668 [2023-09-07 16:39:30,781 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 16:39:30,781 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 16:39:30,781 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 16:39:30,781 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 16:39:30,943 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 16:39:30,944 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 16:39:30,946 INFO][8][curly-red-alterf][T_0] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 16:39:30,951 ERROR][8][curly-red-alterf][T_0] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 16:39:30,951 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:39:31,739 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:39:31,739 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:39:31,739 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:39:31,740 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:39:31,740 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:39:31,740 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:39:31,745 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:39:31,748 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 16:39:31,774 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:39:31,988 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:39:32,158 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:39:32,265 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:39:32,267 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2468 [2023-09-07 16:39:32,269 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:39:32,270 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:39:32,270 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:39:32,271 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:39:32,271 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:39:32,279 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:39:32,282 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:39:32.282381 [2023-09-07 16:39:32,459 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:39:32,496 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:39:32,533 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:39:32,535 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:39:32,543 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:39:32,543 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694104772.492186, 'te': 1694104772.5018263, 'initial_status': True} ====================================================================================================  [2023-09-07 16:39:32,545 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:39:32.545225 [2023-09-07 16:39:32,549 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:39:32,550 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:39:32,826 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104764, schain_finish_ts: 1692377364 [2023-09-07 16:39:33,526 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:39:33,526 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:39:33,528 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:39:33.528720 [2023-09-07 16:39:33,539 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:39:33,539 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:39:33,548 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:39:33,554 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694104773.5368314, 'te': 1694104773.5542364, 'initial_status': True} ====================================================================================================  [2023-09-07 16:39:33,556 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:39:33.556635 [2023-09-07 16:39:33,561 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:39:33,561 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:39:33,997 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:39:33,997 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:39:33,998 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:39:33,998 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:39:33,998 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:39:33,998 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:39:34,000 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:39:34,002 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 16:39:34,038 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:39:34,265 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:39:34,444 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:39:34,582 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:39:34,583 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2452 [2023-09-07 16:39:34,584 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:39:34,584 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:39:34,584 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:39:34,585 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:39:34,585 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:39:34,592 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:39:34,594 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:39:34.594139 [2023-09-07 16:39:34,890 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:39:34,923 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:39:34,960 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:39:34,964 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:39:34,987 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:39:34,987 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694104774.921686, 'te': 1694104774.9286847, 'initial_status': True} ====================================================================================================  [2023-09-07 16:39:34,989 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:39:34.989775 [2023-09-07 16:39:34,995 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:39:34,995 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:39:35,782 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:39:35,782 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:39:35,787 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:39:35.787022 [2023-09-07 16:39:35,797 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:39:35,798 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:39:35,809 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:39:35,813 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694104775.7952726, 'te': 1694104775.8131225, 'initial_status': True} ====================================================================================================  [2023-09-07 16:39:35,815 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:39:35.815028 [2023-09-07 16:39:35,819 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:39:35,819 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:39:39,157 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104776, schain_finish_ts: 1693412520 [2023-09-07 16:39:40,292 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:39:40,292 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:39:40,292 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:39:40,292 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:39:40,292 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:39:40,293 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:39:40,294 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:39:40,296 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 16:39:40,330 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:39:40,599 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:39:40,882 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:39:41,004 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:39:41,005 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2469 [2023-09-07 16:39:41,005 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:39:41,006 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:39:41,006 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:39:41,006 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:39:41,007 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:39:41,012 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:39:41,014 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:39:41.014827 [2023-09-07 16:39:41,273 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:39:41,300 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:39:41,329 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:39:41,331 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:39:41,339 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:39:41,339 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694104781.2991858, 'te': 1694104781.3067746, 'initial_status': True} ====================================================================================================  [2023-09-07 16:39:41,341 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:39:41.341304 [2023-09-07 16:39:41,346 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:39:41,347 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:39:42,088 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:39:42,088 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:39:42,090 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:39:42.090557 [2023-09-07 16:39:42,097 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:39:42,097 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:39:42,104 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:39:42,109 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694104782.095555, 'te': 1694104782.1088958, 'initial_status': True} ====================================================================================================  [2023-09-07 16:39:42,110 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:39:42.110797 [2023-09-07 16:39:42,115 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:39:42,115 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:39:44,217 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 16:39:44,218 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 16:39:45,243 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104776, schain_finish_ts: 0 [2023-09-07 16:39:45,517 INFO][1][MainProcess][MainThread] - core.schains.cleaner:65 - Cleaner process started [2023-09-07 16:39:45,545 INFO][30434][cleaner][MainThread] - core.schains.cleaner:110 - Cleaner procedure started. [2023-09-07 16:39:46,377 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:39:46,378 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:39:46,378 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:40 - Task submissive-well-off-rotanev-config finished with None [2023-09-07 16:39:46,378 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:42 - Running task submissive-well-off-rotanev-config [2023-09-07 16:39:46,378 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 16:39:48,158 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': False} [2023-09-07 16:39:48,159 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:39:48,161 INFO][24912][submissive-well-off-rotanev][T_0] - web.models.schain:152 - Changing monitor_last_seen for submissive-well-off-rotanev to 2023-09-07 16:39:48.161258 [2023-09-07 16:39:48,166 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.action:190 - Creating new upstream_config rotation_id: 8, stream: regression-2.2 [2023-09-07 16:39:48,168 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.config.main:49 - Generating sChain config for submissive-well-off-rotanev [2023-09-07 16:39:51,131 INFO][30434][cleaner][MainThread] - core.schains.cleaner:116 - sChains on contracts: ['portly-passionate-sirius', 'curly-red-alterf', 'howling-hoedus', 'green-giddy-denebola', 'wan-tasty-kitalpha', '', 'submissive-well-off-rotanev'] sChains on node: ['curly-red-alterf', 'green-giddy-denebola', 'howling-hoedus', 'portly-passionate-sirius', 'submissive-well-off-rotanev', 'wan-tasty-kitalpha'] [2023-09-07 16:39:51,132 INFO][30434][cleaner][MainThread] - core.schains.cleaner:132 - Cleanup procedure finished [2023-09-07 16:39:51,136 INFO][1][MainProcess][MainThread] - core.schains.cleaner:67 - Cleaner process is joined. [2023-09-07 16:39:51,136 INFO][1][MainProcess][MainThread] - core.schains.cleaner:68 - Terminating the process [2023-09-07 16:39:51,137 INFO][1][MainProcess][MainThread] - __main__:70 - Sleeping for 90s after run_cleaner [2023-09-07 16:39:55,930 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104788, schain_finish_ts: 1692377364 [2023-09-07 16:39:57,083 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:39:57,083 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:39:57,083 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:39:57,083 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:39:57,084 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:39:57,084 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:39:57,085 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:39:57,086 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 16:39:57,109 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:39:57,358 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:39:57,647 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:39:57,812 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:39:57,814 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2453 [2023-09-07 16:39:57,815 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:39:57,815 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:39:57,816 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:39:57,817 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:39:57,817 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:39:57,831 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:39:57,837 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:39:57.837651 [2023-09-07 16:39:58,210 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:39:58,248 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:39:58,312 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:39:58,315 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:39:58,328 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:39:58,328 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694104798.2440712, 'te': 1694104798.2618434, 'initial_status': True} ====================================================================================================  [2023-09-07 16:39:58,331 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:39:58.331777 [2023-09-07 16:39:58,338 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:39:58,339 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:39:58,996 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:39:58,996 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:39:59,000 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:39:59.000108 [2023-09-07 16:39:59,011 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:39:59,012 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:39:59,020 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:39:59,025 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694104799.008227, 'te': 1694104799.0248466, 'initial_status': True} ====================================================================================================  [2023-09-07 16:39:59,028 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:39:59.028612 [2023-09-07 16:39:59,033 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:39:59,034 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:40:00,400 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104788, schain_finish_ts: 1693742304 [2023-09-07 16:40:01,538 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:40:01,539 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:40:01,539 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:40:01,539 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:40:01,539 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:40:01,540 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 16:40:01,567 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:40:01,762 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:01,927 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:02,037 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:40:02,038 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2475 [2023-09-07 16:40:02,038 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:40:02,039 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:40:02,039 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:40:02,039 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:40:02,039 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:40:02,046 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:40:02,048 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:40:02.048760 [2023-09-07 16:40:02,220 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:02,258 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:40:02,289 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:40:02,292 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:40:02,300 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:40:02,301 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694104802.2567306, 'te': 1694104802.263788, 'initial_status': True} ====================================================================================================  [2023-09-07 16:40:02,302 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:40:02.302758 [2023-09-07 16:40:02,308 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:40:02,309 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:40:03,614 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 16:40:03,799 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 16:40:03,800 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 16:40:03,801 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 16:40:03,801 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 16:40:03,988 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 16:40:03,989 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 16:40:03,990 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 16:40:03,997 ERROR][11][wan-tasty-kitalpha][T_0] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 16:40:03,997 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:40:04,370 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104800, schain_finish_ts: 1693742004 [2023-09-07 16:40:04,696 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104800, schain_finish_ts: 1692637668 [2023-09-07 16:40:05,526 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:40:05,526 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:40:05,527 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:40:05,529 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 16:40:05,530 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 16:40:05,531 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:40:05,535 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:40:05,536 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 16:40:05,580 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:40:05,863 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:05,902 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:40:05,902 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:40:05,903 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:40:05,903 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:40:05,903 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:40:05,903 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:40:05,907 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:40:05,908 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 16:40:05,940 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:40:06,115 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:06,272 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:40:06,272 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104800, schain_finish_ts: 1693412520 [2023-09-07 16:40:06,273 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2468 [2023-09-07 16:40:06,274 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:40:06,274 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:40:06,274 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:40:06,275 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:40:06,275 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:40:06,283 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:40:06,285 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:40:06.285589 [2023-09-07 16:40:06,419 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:06,581 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:06,608 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:40:06,648 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:40:06,650 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:40:06,657 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:40:06,657 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694104806.6052172, 'te': 1694104806.61537, 'initial_status': True} ====================================================================================================  [2023-09-07 16:40:06,659 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:40:06.659123 [2023-09-07 16:40:06,663 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:40:06,664 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:40:06,784 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:06,891 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:40:06,892 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2469 [2023-09-07 16:40:06,893 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:40:06,893 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:40:06,893 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:40:06,894 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:40:06,896 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:40:06,910 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:40:06,912 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:40:06.912747 [2023-09-07 16:40:07,079 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:07,107 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:40:07,149 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:40:07,154 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:40:07,164 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:40:07,165 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694104807.1053975, 'te': 1694104807.1127656, 'initial_status': True} ====================================================================================================  [2023-09-07 16:40:07,167 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:40:07.167409 [2023-09-07 16:40:07,173 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:40:07,176 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:40:07,337 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 16:40:07,337 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:40:07,339 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:40:07.339945 [2023-09-07 16:40:07,345 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 16:40:07,408 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:40:07,408 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:40:07,408 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:40:07,409 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:40:07,409 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:40:07,409 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:40:07,410 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:40:07,411 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 16:40:07,436 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:40:07,684 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:07,708 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:40:07,708 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:40:07,711 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:40:07.711010 [2023-09-07 16:40:07,720 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:40:07,720 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:40:07,733 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:40:07,740 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694104807.7175388, 'te': 1694104807.7397838, 'initial_status': True} ====================================================================================================  [2023-09-07 16:40:07,742 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:40:07.742679 [2023-09-07 16:40:07,748 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:40:07,748 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:40:07,874 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:07,979 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:40:07,980 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2470 [2023-09-07 16:40:07,981 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:40:07,981 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:40:07,981 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:40:07,981 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:40:07,982 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:40:07,987 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:40:07,990 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:40:07.990145 [2023-09-07 16:40:08,182 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:08,220 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:40:08,247 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:40:08,249 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:40:08,257 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:40:08,260 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694104808.2192004, 'te': 1694104808.2254999, 'initial_status': True} ====================================================================================================  [2023-09-07 16:40:08,263 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:40:08.263824 [2023-09-07 16:40:08,267 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:40:08,268 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:40:09,200 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:40:09,200 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:40:09,224 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:40:09.224426 [2023-09-07 16:40:09,239 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:40:09,239 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:40:09,260 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:40:09,265 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694104809.2373402, 'te': 1694104809.2650995, 'initial_status': True} ====================================================================================================  [2023-09-07 16:40:09,267 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:40:09.267373 [2023-09-07 16:40:09,272 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:40:09,273 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:40:12,314 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104800, schain_finish_ts: 0 [2023-09-07 16:40:13,432 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:40:13,433 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:40:13,433 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 16:40:26,429 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 16:40:26,430 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 16:40:26,467 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104812, schain_finish_ts: 1693742004 [2023-09-07 16:40:27,652 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:40:27,652 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:40:27,652 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:40:27,652 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:40:27,653 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:40:27,653 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 16:40:27,677 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:40:27,981 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:28,147 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:28,286 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:40:28,288 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2469 [2023-09-07 16:40:28,289 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:40:28,289 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:40:28,289 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:40:28,289 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:40:28,290 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:40:28,296 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:40:28,299 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:40:28.299053 [2023-09-07 16:40:28,316 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104812, schain_finish_ts: 1693412520 [2023-09-07 16:40:28,470 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:28,499 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:40:28,536 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:40:28,544 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:40:28,555 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:40:28,555 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694104828.4971006, 'te': 1694104828.503769, 'initial_status': True} ====================================================================================================  [2023-09-07 16:40:28,557 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:40:28.557235 [2023-09-07 16:40:28,562 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:40:28,563 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:40:29,422 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:40:29,422 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:40:29,422 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:40:29,423 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:40:29,423 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:40:29,423 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:40:29,423 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:40:29,424 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 16:40:29,463 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:40:29,695 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:30,034 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:30,152 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:40:30,153 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2471 [2023-09-07 16:40:30,154 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:40:30,154 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:40:30,154 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:40:30,155 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:40:30,155 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:40:30,161 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:40:30,163 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:40:30.163519 [2023-09-07 16:40:30,341 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:30,378 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:40:30,422 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:40:30,424 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:40:30,431 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:40:30,432 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694104830.3761907, 'te': 1694104830.3840804, 'initial_status': True} ====================================================================================================  [2023-09-07 16:40:30,434 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:40:30.434070 [2023-09-07 16:40:30,438 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:40:30,439 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:40:31,032 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104812, schain_finish_ts: 1692377364 [2023-09-07 16:40:31,242 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:40:31,242 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:40:31,248 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:40:31.248737 [2023-09-07 16:40:31,265 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:40:31,265 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:40:31,272 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:40:31,277 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694104831.25848, 'te': 1694104831.2774196, 'initial_status': True} ====================================================================================================  [2023-09-07 16:40:31,279 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:40:31.279818 [2023-09-07 16:40:31,285 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:40:31,285 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:40:32,159 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:40:32,159 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:40:32,159 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:40:32,160 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:40:32,160 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:40:32,160 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:40:32,162 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:40:32,163 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 16:40:32,189 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:40:32,409 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:32,578 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:32,681 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:40:32,681 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2454 [2023-09-07 16:40:32,682 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:40:32,682 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:40:32,683 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:40:32,683 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:40:32,683 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:40:32,689 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:40:32,691 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:40:32.691441 [2023-09-07 16:40:32,842 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:32,869 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:40:32,905 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:40:32,907 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:40:32,915 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:40:32,916 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694104832.8673475, 'te': 1694104832.8737636, 'initial_status': True} ====================================================================================================  [2023-09-07 16:40:32,917 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:40:32.917573 [2023-09-07 16:40:32,922 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:40:32,922 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:40:33,959 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:40:33,959 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:40:33,965 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:40:33.965504 [2023-09-07 16:40:33,977 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:40:33,977 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:40:33,990 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:40:33,997 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694104833.9737582, 'te': 1694104833.9973686, 'initial_status': True} ====================================================================================================  [2023-09-07 16:40:34,000 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:40:34.000187 [2023-09-07 16:40:34,007 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:40:34,007 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:40:40,511 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104812, schain_finish_ts: 1693742304 [2023-09-07 16:40:41,622 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:40:41,622 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:40:41,622 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:40:41,622 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 16:40:41,622 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 16:40:41,623 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:40:41,624 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:40:41,626 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 16:40:41,649 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:40:41,900 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:42,176 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:42,312 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:40:42,313 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2476 [2023-09-07 16:40:42,314 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:40:42,314 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:40:42,314 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:40:42,315 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:40:42,315 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:40:42,322 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:40:42,324 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:40:42.324521 [2023-09-07 16:40:42,493 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:42,524 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:40:42,559 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:40:42,561 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:40:42,572 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:40:42,572 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694104842.5230412, 'te': 1694104842.5305471, 'initial_status': True} ====================================================================================================  [2023-09-07 16:40:42,575 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:40:42.575735 [2023-09-07 16:40:42,580 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:40:42,581 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:40:42,833 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104812, schain_finish_ts: 1692637668 [2023-09-07 16:40:43,415 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 16:40:43,416 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:40:43,418 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:40:43.418149 [2023-09-07 16:40:43,423 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 16:40:43,936 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:40:43,936 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:40:43,937 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:40:43,937 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:40:43,937 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:40:43,939 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:40:43,940 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:40:43,944 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 16:40:43,978 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:40:44,261 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:44,411 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:44,519 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:40:44,520 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2470 [2023-09-07 16:40:44,521 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:40:44,521 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:40:44,521 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:40:44,522 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:40:44,522 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:40:44,529 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:40:44,532 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:40:44.532534 [2023-09-07 16:40:44,820 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:40:44,847 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:40:44,885 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:40:44,887 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:40:44,896 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:40:44,897 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694104844.8461967, 'te': 1694104844.85452, 'initial_status': True} ====================================================================================================  [2023-09-07 16:40:44,900 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:40:44.899979 [2023-09-07 16:40:44,908 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:40:44,909 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:40:45,732 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 16:40:45,782 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:40:45,783 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:40:45,785 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:40:45.785216 [2023-09-07 16:40:45,793 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:40:45,793 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:40:45,800 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:40:45,804 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694104845.7915933, 'te': 1694104845.8046455, 'initial_status': True} ====================================================================================================  [2023-09-07 16:40:45,806 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:40:45.806604 [2023-09-07 16:40:45,818 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:40:45,818 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:40:45,908 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 16:40:45,908 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 16:40:45,909 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 16:40:45,909 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 16:40:46,073 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 16:40:46,074 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 16:40:46,076 INFO][8][curly-red-alterf][T_1] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 16:40:46,080 ERROR][8][curly-red-alterf][T_1] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 16:40:46,080 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:40:53,367 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104848, schain_finish_ts: 0 [2023-09-07 16:40:54,570 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:40:54,572 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:40:54,572 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 16:41:00,372 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104848, schain_finish_ts: 1693412520 [2023-09-07 16:41:01,495 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:41:01,495 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:41:01,496 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:41:01,496 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:41:01,497 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:41:01,498 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:41:01,499 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:41:01,499 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 16:41:01,525 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:41:01,604 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104860, schain_finish_ts: 1693742004 [2023-09-07 16:41:01,759 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:41:01,923 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:41:02,045 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:41:02,047 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2472 [2023-09-07 16:41:02,047 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:41:02,047 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:41:02,048 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:41:02,048 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:41:02,048 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:41:02,055 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:41:02,057 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:41:02.057132 [2023-09-07 16:41:02,217 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:41:02,247 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:41:02,276 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:41:02,281 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:41:02,289 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:41:02,289 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694104862.2460363, 'te': 1694104862.2520635, 'initial_status': True} ====================================================================================================  [2023-09-07 16:41:02,291 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:41:02.291425 [2023-09-07 16:41:02,296 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:41:02,297 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:41:02,303 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 16:41:02,303 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 16:41:02,732 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:41:02,733 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:41:02,733 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:41:02,733 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 16:41:02,733 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 16:41:02,734 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:41:02,735 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:41:02,737 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 16:41:02,763 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:41:02,946 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:41:03,098 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:41:03,210 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:41:03,212 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2470 [2023-09-07 16:41:03,213 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:41:03,214 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:41:03,215 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:41:03,215 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:41:03,215 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:41:03,221 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:41:03,224 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:41:03.224126 [2023-09-07 16:41:03,365 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:41:03,390 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:41:03,418 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:41:03,420 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:41:03,427 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:41:03,427 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694104863.3888738, 'te': 1694104863.3951313, 'initial_status': True} ====================================================================================================  [2023-09-07 16:41:03,430 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:41:03.430055 [2023-09-07 16:41:03,434 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:41:03,435 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:41:05,771 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104860, schain_finish_ts: 1692377364 [2023-09-07 16:41:09,305 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:41:09,305 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:41:09,307 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:41:09.307836 [2023-09-07 16:41:09,315 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:41:09,315 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:41:09,322 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:41:09,327 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694104869.313556, 'te': 1694104869.326839, 'initial_status': True} ====================================================================================================  [2023-09-07 16:41:09,328 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:41:09.328744 [2023-09-07 16:41:09,333 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:41:09,334 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:41:14,665 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 16:41:14,666 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:41:14,668 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:41:14.668490 [2023-09-07 16:41:14,674 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 16:41:15,456 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:41:15,456 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:41:15,456 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:41:15,456 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:41:15,456 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:41:15,457 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:41:15,458 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:41:15,458 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 16:41:15,485 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:41:15,688 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:41:15,823 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:41:15,927 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:41:15,928 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2455 [2023-09-07 16:41:15,928 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:41:15,928 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:41:15,929 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:41:15,929 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:41:15,930 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:41:15,937 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:41:15,939 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:41:15.939188 [2023-09-07 16:41:16,158 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:41:16,185 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:41:16,213 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:41:16,214 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:41:16,225 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:41:16,226 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694104876.183956, 'te': 1694104876.1905937, 'initial_status': True} ====================================================================================================  [2023-09-07 16:41:16,228 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:41:16.227977 [2023-09-07 16:41:16,232 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:41:16,233 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:41:19,766 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104872, schain_finish_ts: 1693742304 [2023-09-07 16:41:20,460 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104872, schain_finish_ts: 1692637668 [2023-09-07 16:41:21,201 INFO][1][MainProcess][MainThread] - core.schains.process_manager:57 - Process manager started [2023-09-07 16:41:23,013 INFO][1][MainProcess][MainThread] - core.schains.notifications:35 - Node account has 406869999689179237 WEI [2023-09-07 16:41:23,014 INFO][1][MainProcess][MainThread] - tools.notifications.messages:188 - Saving new balance state 3005 1 [2023-09-07 16:41:23,015 INFO][1][MainProcess][MainThread] - core.schains.process_manager:95 - Fetching schains to monitor... [2023-09-07 16:41:23,461 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104872, schain_finish_ts: 1693412520 [2023-09-07 16:41:27,809 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:41:27,809 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:41:27,809 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:41:27,810 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:41:27,810 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:41:27,810 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 16:41:27,836 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:41:28,016 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104884, schain_finish_ts: 0 [2023-09-07 16:41:28,023 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:41:28,190 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:41:28,218 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:41:28,218 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:41:28,220 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:41:28.220852 [2023-09-07 16:41:28,231 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:41:28,232 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:41:28,238 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:41:28,242 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694104888.2299774, 'te': 1694104888.2427576, 'initial_status': True} ====================================================================================================  [2023-09-07 16:41:28,244 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:41:28.244710 [2023-09-07 16:41:28,248 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:41:28,249 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:41:28,333 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:41:28,334 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2477 [2023-09-07 16:41:28,335 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:41:28,335 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:41:28,336 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:41:28,336 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:41:28,337 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:41:28,348 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:41:28,352 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:41:28.352078 [2023-09-07 16:41:28,521 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:41:28,548 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:41:28,584 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:41:28,586 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:41:28,594 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:41:28,595 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694104888.5465505, 'te': 1694104888.5542097, 'initial_status': True} ====================================================================================================  [2023-09-07 16:41:28,597 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:41:28.597503 [2023-09-07 16:41:28,602 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:41:28,603 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:41:33,182 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:41:33,182 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:41:33,182 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:41:33,183 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:41:33,183 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:41:33,183 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:41:33,186 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:41:33,189 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 16:41:33,214 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:41:33,419 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:41:33,569 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:41:33,681 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:41:33,683 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2471 [2023-09-07 16:41:33,684 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:41:33,684 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:41:33,684 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:41:33,685 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:41:33,685 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:41:33,693 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:41:33,696 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:41:33.695992 [2023-09-07 16:41:33,866 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:41:33,897 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:41:33,931 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:41:33,933 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:41:33,942 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:41:33,943 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694104893.8913064, 'te': 1694104893.9033976, 'initial_status': True} ====================================================================================================  [2023-09-07 16:41:33,944 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:41:33.944709 [2023-09-07 16:41:33,949 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:41:33,950 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:41:35,242 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:41:35,243 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:41:35,243 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:41:35,243 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:41:35,243 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:41:35,244 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:41:35,246 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:41:35,247 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 16:41:35,273 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:41:35,457 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:41:35,692 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:41:35,786 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:41:35,787 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2473 [2023-09-07 16:41:35,788 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:41:35,788 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:41:35,788 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:41:35,789 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:41:35,789 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:41:35,795 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:41:35,796 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:41:35.796959 [2023-09-07 16:41:35,934 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:41:35,961 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:41:35,989 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:41:35,991 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:41:35,998 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:41:35,999 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694104895.9601731, 'te': 1694104895.9669611, 'initial_status': True} ====================================================================================================  [2023-09-07 16:41:36,000 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:41:36.000633 [2023-09-07 16:41:36,005 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:41:36,005 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:41:51,937 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:41:51,938 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:41:51,938 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 16:41:59,822 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104908, schain_finish_ts: 1693742004 [2023-09-07 16:42:00,892 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:42:00,892 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:42:00,895 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:42:00.895090 [2023-09-07 16:42:00,939 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:42:00,940 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:42:00,951 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:42:00,957 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694104920.9365723, 'te': 1694104920.9574957, 'initial_status': True} ====================================================================================================  [2023-09-07 16:42:00,959 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:42:00.959432 [2023-09-07 16:42:00,969 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:42:00,969 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:42:03,243 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104908, schain_finish_ts: 1692377364 [2023-09-07 16:42:11,169 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:42:11,170 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:42:11,172 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:42:11.172833 [2023-09-07 16:42:11,183 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:42:11,184 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:42:11,190 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:42:11,196 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694104931.1793203, 'te': 1694104931.1962814, 'initial_status': True} ====================================================================================================  [2023-09-07 16:42:11,198 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:42:11.198287 [2023-09-07 16:42:11,205 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:42:11,205 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:42:11,212 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104908, schain_finish_ts: 1693742304 [2023-09-07 16:42:17,982 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:42:17,982 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:42:17,982 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:42:17,983 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:42:17,983 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:42:17,984 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 16:42:18,019 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:42:18,373 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:42:18,509 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:42:18,651 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:42:18,652 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2471 [2023-09-07 16:42:18,653 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:42:18,653 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:42:18,653 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:42:18,654 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:42:18,654 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:42:18,663 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:42:18,666 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:42:18.666134 [2023-09-07 16:42:18,831 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:42:18,859 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:42:18,889 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:42:18,892 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:42:18,900 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:42:18,900 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694104938.8579476, 'te': 1694104938.8643289, 'initial_status': True} ====================================================================================================  [2023-09-07 16:42:18,902 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:42:18.902610 [2023-09-07 16:42:18,906 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:42:18,907 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:42:21,780 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104932, schain_finish_ts: 1693412520 [2023-09-07 16:42:29,021 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104932, schain_finish_ts: 1692637668 [2023-09-07 16:42:29,805 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:42:29,805 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:42:29,805 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:42:29,806 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:42:29,806 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:42:29,806 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:42:29,808 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:42:29,811 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 16:42:29,835 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:42:30,121 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:42:30,309 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104932, schain_finish_ts: 0 [2023-09-07 16:42:30,341 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:42:30,473 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:42:30,475 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2456 [2023-09-07 16:42:30,476 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:42:30,476 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:42:30,476 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:42:30,477 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:42:30,477 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:42:30,494 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:42:30,499 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:42:30.499766 [2023-09-07 16:42:30,690 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:42:30,717 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:42:30,745 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:42:30,747 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:42:30,754 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:42:30,755 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694104950.7156909, 'te': 1694104950.7222803, 'initial_status': True} ====================================================================================================  [2023-09-07 16:42:30,757 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:42:30.757022 [2023-09-07 16:42:30,761 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:42:30,762 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:42:41,457 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:42:41,459 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:42:41,459 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:42:41,459 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:42:41,459 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:42:41,461 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:42:41,469 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:42:41,470 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 16:42:41,504 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:42:41,752 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:42:41,786 INFO][1][MainProcess][MainThread] - core.schains.process_manager:109 - Get leaving_history for node ... [2023-09-07 16:42:41,926 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:42:42,070 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:42:42,072 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2474 [2023-09-07 16:42:42,072 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:42:42,073 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:42:42,073 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:42:42,073 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:42:42,074 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:42:42,080 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:42:42,082 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:42:42.082375 [2023-09-07 16:42:42,148 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:42:42,148 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:42:42,148 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:42:42,149 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:42:42,149 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:42:42,149 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:42:42,151 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:42:42,155 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 16:42:42,184 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:42:42,239 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:42:42,270 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:42:42,305 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:42:42,309 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:42:42,319 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:42:42,319 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694104962.2681642, 'te': 1694104962.2743604, 'initial_status': True} ====================================================================================================  [2023-09-07 16:42:42,321 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:42:42.321399 [2023-09-07 16:42:42,326 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:42:42,326 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:42:42,391 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:42:42,535 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:42:42,648 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:42:42,651 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2472 [2023-09-07 16:42:42,652 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:42:42,652 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:42:42,653 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:42:42,654 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:42:42,654 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:42:42,661 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:42:42,665 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:42:42.665115 [2023-09-07 16:42:43,023 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:42:43,053 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:42:43,090 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:42:43,092 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:42:43,099 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:42:43,100 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694104963.0512476, 'te': 1694104963.0586774, 'initial_status': True} ====================================================================================================  [2023-09-07 16:42:43,101 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:42:43.101978 [2023-09-07 16:42:43,106 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:42:43,107 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:42:43,496 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:42:43,496 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:42:43,496 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:42:43,497 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:42:43,497 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:42:43,497 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 16:42:43,520 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:42:43,819 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:42:43,972 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:42:44,077 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:42:44,078 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2478 [2023-09-07 16:42:44,079 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:42:44,079 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:42:44,080 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:42:44,080 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:42:44,080 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:42:44,086 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:42:44,088 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:42:44.088281 [2023-09-07 16:42:44,227 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:42:44,252 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:42:44,285 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:42:44,287 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:42:44,295 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:42:44,295 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694104964.2505424, 'te': 1694104964.2568142, 'initial_status': True} ====================================================================================================  [2023-09-07 16:42:44,297 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:42:44.297339 [2023-09-07 16:42:44,302 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:42:44,302 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:42:48,223 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:42:48,223 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:42:48,229 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:42:48.229081 [2023-09-07 16:42:48,241 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:42:48,241 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:42:48,250 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:42:48,256 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694104968.2398438, 'te': 1694104968.2555618, 'initial_status': True} ====================================================================================================  [2023-09-07 16:42:48,259 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:42:48.259860 [2023-09-07 16:42:48,270 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:42:48,270 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:42:51,025 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:42:51,025 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:42:51,026 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 16:42:57,285 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:42:57,286 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:42:57,289 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:42:57.289887 [2023-09-07 16:42:57,303 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:42:57,303 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:42:57,309 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:42:57,315 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694104977.3016078, 'te': 1694104977.3149817, 'initial_status': True} ====================================================================================================  [2023-09-07 16:42:57,316 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:42:57.316931 [2023-09-07 16:42:57,323 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:42:57,323 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:42:57,507 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104956, schain_finish_ts: 1692377616 [2023-09-07 16:42:59,742 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104956, schain_finish_ts: 1693742004 [2023-09-07 16:43:00,204 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:43:00,204 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:43:00,206 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:43:00.206073 [2023-09-07 16:43:00,214 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:43:00,214 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:43:00,220 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:43:00,226 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694104980.2124128, 'te': 1694104980.225982, 'initial_status': True} ====================================================================================================  [2023-09-07 16:43:00,228 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:43:00.228218 [2023-09-07 16:43:00,233 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:43:00,233 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:43:06,540 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104980, schain_finish_ts: 1692377364 [2023-09-07 16:43:10,979 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:43:10,980 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:43:10,980 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:43:10,980 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:43:10,980 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:43:10,981 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 16:43:11,003 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:43:11,220 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:43:11,375 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:43:11,471 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:43:11,471 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2472 [2023-09-07 16:43:11,472 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:43:11,472 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:43:11,473 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:43:11,473 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:43:11,474 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:43:11,480 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:43:11,482 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:43:11.482457 [2023-09-07 16:43:11,649 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:43:11,675 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:43:11,703 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:43:11,705 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:43:11,711 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:43:11,712 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694104991.6741555, 'te': 1694104991.6801095, 'initial_status': True} ====================================================================================================  [2023-09-07 16:43:11,713 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:43:11.713866 [2023-09-07 16:43:11,719 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:43:11,719 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:43:12,697 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:43:12,697 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:43:12,697 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:43:12,697 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:43:12,698 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:43:12,698 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:43:12,701 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:43:12,702 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 16:43:12,733 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:43:12,911 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:43:13,143 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:43:13,238 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:43:13,239 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2457 [2023-09-07 16:43:13,242 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:43:13,242 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:43:13,242 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:43:13,242 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:43:13,243 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:43:13,249 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:43:13,251 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:43:13.251728 [2023-09-07 16:43:13,386 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:43:13,417 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:43:13,446 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:43:13,448 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:43:13,455 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:43:13,455 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694104993.416288, 'te': 1694104993.4242105, 'initial_status': True} ====================================================================================================  [2023-09-07 16:43:13,457 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:43:13.457159 [2023-09-07 16:43:13,461 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:43:13,462 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:43:27,771 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694104980, schain_finish_ts: 1693412520 [2023-09-07 16:43:33,347 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105004, schain_finish_ts: 0 [2023-09-07 16:43:33,785 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105004, schain_finish_ts: 1693742304 [2023-09-07 16:43:33,885 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105004, schain_finish_ts: 0 [2023-09-07 16:43:36,002 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105004, schain_finish_ts: 1692637668 [2023-09-07 16:43:36,247 INFO][1][MainProcess][MainThread] - core.schains.process_manager:117 - Got leaving sChains for the node: [] [2023-09-07 16:43:36,248 INFO][1][MainProcess][MainThread] - core.schains.process_manager:101 -  ==================================================================================================== Monitoring sChains Node ID: 16 sChains on node: [{'name': 'portly-passionate-sirius', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666281384, 'startBlock': 7803380, 'deposit': 0, 'index': 1, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x0000017723ca2', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'curly-red-alterf', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1676058936, 'startBlock': 8469646, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000001fe6cdfa', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'howling-hoedus', 'mainnetOwner': '0x1026467C91fdA40aCc7e6935d1E52f67274FD669', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666181400, 'startBlock': 7796660, 'deposit': 0, 'index': 0, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0xe5a600f4c2608', 'multitransactionMode': False, 'thresholdEncryption': False, 'active': True}, {'name': 'green-giddy-denebola', 'mainnetOwner': '0x8e835d5d9E28ffD259E8Fa8988Cacfc333A582B4', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674565164, 'startBlock': 8368049, 'deposit': 0, 'index': 4, 'generation': 1, 'originator': '0xd2001DAb6898127Be2F167B548691C87251D13C3', 'chainId': '0x00000585eb4b1', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'wan-tasty-kitalpha', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674568392, 'startBlock': 8368271, 'deposit': 0, 'index': 5, 'generation': 1, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x00000152e7be9', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'submissive-well-off-rotanev', 'mainnetOwner': '0x487360568fE5Ee4AE5c75782D5F63e9935B0E512', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1692361104, 'startBlock': 9539053, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000003791fa10', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}] Number of sChains on node: 6 Empty sChain structs: 1 ====================================================================================================  [2023-09-07 16:43:41,631 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:43:41,631 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain portly-passionate-sirius - Process is running: PID = 7 [2023-09-07 16:43:45,581 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:43:45,581 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain curly-red-alterf - Process is running: PID = 8 [2023-09-07 16:43:45,702 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:43:45,703 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:43:45,703 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:43:45,703 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:43:45,703 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:43:45,704 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 16:43:45,724 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:43:45,966 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:43:46,157 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:43:46,292 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:43:46,293 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2479 [2023-09-07 16:43:46,294 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:43:46,294 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:43:46,295 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:43:46,295 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:43:46,296 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:43:46,304 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:43:46,308 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:43:46.308122 [2023-09-07 16:43:46,525 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:43:46,556 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:43:46,590 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:43:46,593 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:43:46,602 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:43:46,602 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694105026.5542295, 'te': 1694105026.5606842, 'initial_status': True} ====================================================================================================  [2023-09-07 16:43:46,604 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:43:46.604861 [2023-09-07 16:43:46,609 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:43:46,610 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:43:50,791 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:43:50,792 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain howling-hoedus - Process is running: PID = 9 [2023-09-07 16:43:52,715 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:43:52,715 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:43:52,715 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 16:43:52,762 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:43:52,762 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:43:52,762 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:43:52,762 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:43:52,763 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:43:52,763 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:43:52,764 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:43:52,765 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 16:43:52,796 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:43:53,146 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:43:53,309 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:43:53,419 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:43:53,420 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2475 [2023-09-07 16:43:53,421 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:43:53,421 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:43:53,421 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:43:53,421 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:43:53,422 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:43:53,428 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:43:53,430 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:43:53.430927 [2023-09-07 16:43:53,611 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:43:53,640 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:43:53,676 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:43:53,681 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:43:53,693 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:43:53,695 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694105033.6390967, 'te': 1694105033.6452808, 'initial_status': True} ====================================================================================================  [2023-09-07 16:43:53,698 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:43:53.698345 [2023-09-07 16:43:53,708 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:43:53,718 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:43:55,036 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:43:55,037 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain green-giddy-denebola - Process is running: PID = 10 [2023-09-07 16:44:00,379 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:44:00,379 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:44:00,381 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:44:00.381544 [2023-09-07 16:44:00,388 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:44:00,388 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:44:00,393 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:44:00,401 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694105040.3868635, 'te': 1694105040.4011135, 'initial_status': True} ====================================================================================================  [2023-09-07 16:44:00,403 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:44:00.403130 [2023-09-07 16:44:00,408 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:44:00,409 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:44:05,818 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:44:05,818 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:44:05,818 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:44:05,818 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:44:05,818 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:44:05,819 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:44:05,819 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:44:05,819 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 16:44:05,844 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:44:06,059 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:44:06,111 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:44:06,111 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain wan-tasty-kitalpha - Process is running: PID = 11 [2023-09-07 16:44:06,293 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:44:06,421 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:44:06,422 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2473 [2023-09-07 16:44:06,423 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:44:06,423 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:44:06,423 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:44:06,424 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:44:06,424 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:44:06,430 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:44:06,431 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:44:06.431965 [2023-09-07 16:44:06,569 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:44:06,594 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:44:06,624 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:44:06,626 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:44:06,633 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:44:06,633 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694105046.5927193, 'te': 1694105046.6011586, 'initial_status': True} ====================================================================================================  [2023-09-07 16:44:06,635 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:44:06.635085 [2023-09-07 16:44:06,639 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:44:06,640 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:44:10,728 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105040, schain_finish_ts: 1692377364 [2023-09-07 16:44:10,930 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:44:10,930 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain submissive-well-off-rotanev - Process is running: PID = 24912 [2023-09-07 16:44:10,930 INFO][1][MainProcess][MainThread] - core.schains.process_manager:88 - Process manager procedure finished [2023-09-07 16:44:10,930 INFO][1][MainProcess][MainThread] - __main__:65 - Sleeping for 90s after run_process_manager [2023-09-07 16:44:10,974 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105040, schain_finish_ts: 1693742004 [2023-09-07 16:44:35,444 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105064, schain_finish_ts: 1693742304 [2023-09-07 16:44:35,491 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:44:35,491 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:44:35,491 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:44:35,492 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:44:35,492 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:44:35,492 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:44:35,496 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:44:35,498 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 16:44:35,519 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:44:35,598 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:44:35,598 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:44:35,601 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:44:35.601595 [2023-09-07 16:44:35,610 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:44:35,611 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:44:35,618 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:44:35,624 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694105075.6083803, 'te': 1694105075.6242616, 'initial_status': True} ====================================================================================================  [2023-09-07 16:44:35,627 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:44:35.627315 [2023-09-07 16:44:35,632 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:44:35,632 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:44:35,705 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:44:35,842 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:44:35,959 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:44:35,962 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2458 [2023-09-07 16:44:35,964 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:44:35,964 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:44:35,965 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:44:35,965 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:44:35,966 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:44:35,973 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:44:35,975 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:44:35.975679 [2023-09-07 16:44:36,130 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:44:36,157 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:44:36,189 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:44:36,191 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:44:36,199 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:44:36,199 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694105076.1556883, 'te': 1694105076.1622922, 'initial_status': True} ====================================================================================================  [2023-09-07 16:44:36,204 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:44:36.204398 [2023-09-07 16:44:36,211 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:44:36,211 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:44:37,606 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105064, schain_finish_ts: 1692637668 [2023-09-07 16:44:40,456 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105076, schain_finish_ts: 0 [2023-09-07 16:44:46,837 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:44:46,837 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:44:46,838 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:44:46,838 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:44:46,838 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:44:46,838 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 16:44:46,867 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:44:47,176 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:44:47,311 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:44:47,452 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:44:47,453 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2473 [2023-09-07 16:44:47,454 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:44:47,454 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:44:47,454 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:44:47,455 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:44:47,455 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:44:47,467 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:44:47,470 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105076, schain_finish_ts: 1693412520 [2023-09-07 16:44:47,471 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:44:47.471142 [2023-09-07 16:44:47,666 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:44:47,693 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:44:47,722 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:44:47,725 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:44:47,732 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:44:47,733 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694105087.6912963, 'te': 1694105087.697896, 'initial_status': True} ====================================================================================================  [2023-09-07 16:44:47,734 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:44:47.734694 [2023-09-07 16:44:47,738 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:44:47,739 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:44:52,899 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:44:52,899 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:44:52,899 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:44:52,899 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:44:52,900 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:44:52,900 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 16:44:52,923 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:44:53,268 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:44:53,392 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:44:53,507 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:44:53,508 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2480 [2023-09-07 16:44:53,509 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:44:53,509 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:44:53,509 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:44:53,509 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:44:53,509 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:44:53,515 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:44:53,517 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:44:53.517303 [2023-09-07 16:44:53,650 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:44:53,681 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:44:53,707 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:44:53,709 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:44:53,716 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:44:53,716 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694105093.6802025, 'te': 1694105093.6858754, 'initial_status': True} ====================================================================================================  [2023-09-07 16:44:53,718 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:44:53.718416 [2023-09-07 16:44:53,724 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:44:53,725 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:45:02,608 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:45:02,608 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:45:02,611 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:45:02.611184 [2023-09-07 16:45:02,619 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:45:02,620 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:45:02,627 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:45:02,632 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694105102.6181889, 'te': 1694105102.6319456, 'initial_status': True} ====================================================================================================  [2023-09-07 16:45:02,634 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:45:02.633985 [2023-09-07 16:45:02,640 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:45:02,640 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:45:19,652 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:45:19,652 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:45:19,653 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:45:19,653 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:45:19,653 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:45:19,653 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:45:19,655 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:45:19,657 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 16:45:19,695 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:45:19,874 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:45:20,107 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:45:20,225 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:45:20,227 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2474 [2023-09-07 16:45:20,227 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:45:20,228 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:45:20,228 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:45:20,228 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:45:20,228 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:45:20,234 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:45:20,236 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:45:20.236198 [2023-09-07 16:45:20,365 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:45:20,391 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:45:20,431 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:45:20,433 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:45:20,442 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:45:20,443 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694105120.3898604, 'te': 1694105120.3960984, 'initial_status': True} ====================================================================================================  [2023-09-07 16:45:20,445 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:45:20.445200 [2023-09-07 16:45:20,450 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:45:20,451 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:45:21,777 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:45:21,777 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:45:21,777 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 16:45:23,358 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:45:23,358 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:45:23,360 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:45:23.360638 [2023-09-07 16:45:23,367 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:45:23,368 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:45:23,375 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:45:23,380 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694105123.3664834, 'te': 1694105123.3800192, 'initial_status': True} ====================================================================================================  [2023-09-07 16:45:23,382 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:45:23.382416 [2023-09-07 16:45:23,391 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:45:23,391 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:45:25,089 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105112, schain_finish_ts: 1692377364 [2023-09-07 16:45:26,303 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:45:26,304 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:45:26,305 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:45:26,305 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:45:26,306 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:45:26,306 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:45:26,308 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:45:26,308 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 16:45:26,338 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:45:26,538 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:45:26,683 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:45:26,790 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:45:26,792 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2476 [2023-09-07 16:45:26,792 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:45:26,793 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:45:26,793 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:45:26,793 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:45:26,794 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:45:26,799 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:45:26,801 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:45:26.801625 [2023-09-07 16:45:27,038 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:45:27,066 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:45:27,093 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:45:27,095 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:45:27,101 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:45:27,102 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694105127.064549, 'te': 1694105127.0706234, 'initial_status': True} ====================================================================================================  [2023-09-07 16:45:27,103 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:45:27.103741 [2023-09-07 16:45:27,107 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:45:27,108 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:45:41,021 INFO][1][MainProcess][MainThread] - core.schains.cleaner:65 - Cleaner process started [2023-09-07 16:45:41,052 INFO][30465][cleaner][MainThread] - core.schains.cleaner:110 - Cleaner procedure started. [2023-09-07 16:45:43,996 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105136, schain_finish_ts: 1693742004 [2023-09-07 16:45:47,475 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105136, schain_finish_ts: 1693742304 [2023-09-07 16:45:52,348 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:45:52,348 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:45:52,349 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:45:52,349 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:45:52,349 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:45:52,349 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:45:52,351 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:45:52,351 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 16:45:52,373 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:45:52,558 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:45:52,786 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:45:52,888 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:45:52,889 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2459 [2023-09-07 16:45:52,890 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:45:52,890 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:45:52,891 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:45:52,891 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:45:52,891 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:45:52,897 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:45:52,900 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:45:52.899993 [2023-09-07 16:45:53,030 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:45:53,055 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:45:53,084 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:45:53,086 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:45:53,094 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:45:53,094 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694105153.0535588, 'te': 1694105153.0595098, 'initial_status': True} ====================================================================================================  [2023-09-07 16:45:53,096 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:45:53.096519 [2023-09-07 16:45:53,100 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:45:53,101 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:45:54,857 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 16:45:55,034 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 16:45:55,035 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 16:45:55,035 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 16:45:55,035 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 16:45:55,211 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 16:45:55,211 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 16:45:55,213 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 16:45:55,218 ERROR][11][wan-tasty-kitalpha][T_1] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 16:45:55,219 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:46:00,866 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:46:00,868 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:46:00,868 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:46:00,868 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:46:00,868 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:46:00,868 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 16:46:00,893 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:46:01,111 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:46:01,253 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:46:01,369 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:46:01,370 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2474 [2023-09-07 16:46:01,371 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:46:01,372 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:46:01,372 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:46:01,373 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:46:01,374 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:46:01,385 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:46:01,388 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:46:01.388504 [2023-09-07 16:46:01,644 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:46:01,670 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:46:01,698 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:46:01,700 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:46:01,707 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:46:01,708 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694105161.66917, 'te': 1694105161.675524, 'initial_status': True} ====================================================================================================  [2023-09-07 16:46:01,709 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:46:01.709779 [2023-09-07 16:46:01,714 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:46:01,714 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:46:02,907 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:46:02,908 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:46:02,912 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:46:02.912819 [2023-09-07 16:46:02,923 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:46:02,924 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:46:02,931 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:46:02,935 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694105162.9220448, 'te': 1694105162.9352365, 'initial_status': True} ====================================================================================================  [2023-09-07 16:46:02,937 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:46:02.937145 [2023-09-07 16:46:02,942 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:46:02,943 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:46:03,371 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:46:03,371 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:46:03,372 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:46:03,372 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 16:46:03,372 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 16:46:03,372 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:46:03,375 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:46:03,376 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 16:46:03,397 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:46:03,626 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:46:03,769 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:46:03,867 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:46:03,868 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2481 [2023-09-07 16:46:03,869 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:46:03,869 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:46:03,869 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:46:03,869 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:46:03,869 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:46:03,875 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:46:03,877 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:46:03.877372 [2023-09-07 16:46:04,035 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:46:04,070 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:46:04,111 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:46:04,114 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:46:04,124 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:46:04,124 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694105164.0688727, 'te': 1694105164.0778503, 'initial_status': True} ====================================================================================================  [2023-09-07 16:46:04,126 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:46:04.126405 [2023-09-07 16:46:04,131 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:46:04,133 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:46:09,198 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105148, schain_finish_ts: 0 [2023-09-07 16:46:19,763 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:46:19,764 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:46:19,768 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:46:19.768410 [2023-09-07 16:46:19,777 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:46:19,777 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:46:19,785 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:46:19,789 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694105179.7756279, 'te': 1694105179.789086, 'initial_status': True} ====================================================================================================  [2023-09-07 16:46:19,791 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:46:19.791295 [2023-09-07 16:46:19,796 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:46:19,796 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:46:34,904 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:46:34,904 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:46:34,904 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 16:46:38,010 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105184, schain_finish_ts: 1692637668 [2023-09-07 16:46:38,126 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105196, schain_finish_ts: 1693412520 [2023-09-07 16:46:44,696 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105196, schain_finish_ts: 1692377364 [2023-09-07 16:46:48,951 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 16:46:48,952 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:46:48,954 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:46:48.954515 [2023-09-07 16:46:48,963 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 16:46:51,268 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105196, schain_finish_ts: 1693742004 [2023-09-07 16:46:52,901 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:46:52,902 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:46:52,904 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:46:52.904077 [2023-09-07 16:46:52,912 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:46:52,912 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:46:52,917 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:46:52,921 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694105212.910717, 'te': 1694105212.921483, 'initial_status': True} ====================================================================================================  [2023-09-07 16:46:52,923 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:46:52.923456 [2023-09-07 16:46:52,927 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:46:52,928 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:47:04,173 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:47:04,173 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:47:04,173 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:47:04,173 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:47:04,174 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:47:04,174 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:47:04,176 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:47:04,179 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 16:47:04,202 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:47:04,380 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:47:04,554 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:47:04,694 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:47:04,695 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2475 [2023-09-07 16:47:04,696 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:47:04,696 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:47:04,696 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:47:04,697 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:47:04,697 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:47:04,703 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:47:04,705 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:47:04.705714 [2023-09-07 16:47:04,854 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:47:04,886 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:47:04,916 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:47:04,918 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:47:04,925 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:47:04,925 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694105224.8847344, 'te': 1694105224.890885, 'initial_status': True} ====================================================================================================  [2023-09-07 16:47:04,927 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:47:04.927376 [2023-09-07 16:47:04,932 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:47:04,933 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:47:05,083 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 16:47:05,085 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 16:47:06,540 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105208, schain_finish_ts: 1693742304 [2023-09-07 16:47:12,071 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105208, schain_finish_ts: 0 [2023-09-07 16:47:12,673 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:47:12,674 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:47:12,674 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:47:12,674 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:47:12,675 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:47:12,675 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:47:12,675 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:47:12,679 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 16:47:12,707 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:47:12,952 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:47:13,106 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:47:13,232 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:47:13,233 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2477 [2023-09-07 16:47:13,234 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:47:13,234 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:47:13,234 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:47:13,234 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:47:13,235 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:47:13,240 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:47:13,242 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:47:13.242902 [2023-09-07 16:47:13,360 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:47:13,361 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:47:13,361 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:47:13,361 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:47:13,361 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:47:13,362 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 16:47:13,387 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:47:13,413 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:47:13,460 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:47:13,495 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:47:13,497 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:47:13,506 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:47:13,506 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694105233.4582784, 'te': 1694105233.4659653, 'initial_status': True} ====================================================================================================  [2023-09-07 16:47:13,508 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:47:13.508403 [2023-09-07 16:47:13,515 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:47:13,520 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:47:13,670 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:47:13,889 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:47:14,004 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:47:14,005 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2475 [2023-09-07 16:47:14,005 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:47:14,006 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:47:14,006 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:47:14,006 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:47:14,007 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:47:14,015 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:47:14,017 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:47:14.017793 [2023-09-07 16:47:14,169 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:47:14,207 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:47:14,246 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:47:14,248 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:47:14,255 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:47:14,256 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694105234.2059345, 'te': 1694105234.2121973, 'initial_status': True} ====================================================================================================  [2023-09-07 16:47:14,258 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:47:14.258283 [2023-09-07 16:47:14,264 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:47:14,265 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:47:14,576 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:47:14,577 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:47:14,577 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:47:14,577 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:47:14,577 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:47:14,578 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:47:14,579 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:47:14,586 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 16:47:14,613 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:47:14,825 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:47:15,017 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:47:15,228 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:47:15,238 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2460 [2023-09-07 16:47:15,241 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:47:15,241 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:47:15,248 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:47:15,248 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:47:15,249 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:47:15,293 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:47:15,295 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:47:15.295382 [2023-09-07 16:47:15,431 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:47:15,456 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:47:15,484 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:47:15,486 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:47:15,492 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:47:15,493 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694105235.455057, 'te': 1694105235.4611673, 'initial_status': True} ====================================================================================================  [2023-09-07 16:47:15,494 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:47:15.494499 [2023-09-07 16:47:15,498 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:47:15,499 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:47:22,872 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:47:22,873 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:47:22,873 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 16:47:28,470 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:47:28,471 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:47:28,471 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:47:28,471 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:47:28,472 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:47:28,472 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 16:47:28,498 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:47:28,975 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:47:29,103 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:47:29,247 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:47:29,255 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2482 [2023-09-07 16:47:29,265 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:47:29,266 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:47:29,266 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:47:29,267 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:47:29,267 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:47:29,274 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:47:29,277 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:47:29.277321 [2023-09-07 16:47:29,477 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:47:29,510 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:47:29,540 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:47:29,542 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:47:29,548 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:47:29,549 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694105249.5087543, 'te': 1694105249.515161, 'initial_status': True} ====================================================================================================  [2023-09-07 16:47:29,550 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:47:29.550789 [2023-09-07 16:47:29,555 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:47:29,556 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:47:31,648 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:47:31,648 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:47:31,652 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:47:31.652139 [2023-09-07 16:47:31,659 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:47:31,659 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:47:31,665 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:47:31,670 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694105251.6576428, 'te': 1694105251.6699758, 'initial_status': True} ====================================================================================================  [2023-09-07 16:47:31,672 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:47:31.672194 [2023-09-07 16:47:31,677 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:47:31,677 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:47:37,998 INFO][30465][cleaner][MainThread] - core.schains.cleaner:116 - sChains on contracts: ['portly-passionate-sirius', 'curly-red-alterf', 'howling-hoedus', 'green-giddy-denebola', 'wan-tasty-kitalpha', '', 'submissive-well-off-rotanev'] sChains on node: ['curly-red-alterf', 'green-giddy-denebola', 'howling-hoedus', 'portly-passionate-sirius', 'submissive-well-off-rotanev', 'wan-tasty-kitalpha'] [2023-09-07 16:47:37,999 INFO][30465][cleaner][MainThread] - core.schains.cleaner:132 - Cleanup procedure finished [2023-09-07 16:47:38,004 INFO][1][MainProcess][MainThread] - core.schains.cleaner:67 - Cleaner process is joined. [2023-09-07 16:47:38,004 INFO][1][MainProcess][MainThread] - core.schains.cleaner:68 - Terminating the process [2023-09-07 16:47:38,004 INFO][1][MainProcess][MainThread] - __main__:70 - Sleeping for 90s after run_cleaner [2023-09-07 16:47:42,715 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:47:42,716 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:47:42,718 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:47:42.718782 [2023-09-07 16:47:42,727 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:47:42,727 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:47:42,735 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:47:42,741 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694105262.7250063, 'te': 1694105262.7411134, 'initial_status': True} ====================================================================================================  [2023-09-07 16:47:42,743 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:47:42.743800 [2023-09-07 16:47:42,748 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:47:42,749 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:47:47,758 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105256, schain_finish_ts: 1692637668 [2023-09-07 16:47:47,925 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:47:47,927 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:47:47,932 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:47:47.932297 [2023-09-07 16:47:47,944 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:47:47,946 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:47:47,952 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:47:47,958 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694105267.9429908, 'te': 1694105267.9578438, 'initial_status': True} ====================================================================================================  [2023-09-07 16:47:47,959 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:47:47.959743 [2023-09-07 16:47:47,964 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:47:47,964 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:48:00,464 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105268, schain_finish_ts: 1693412520 [2023-09-07 16:48:05,407 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105280, schain_finish_ts: 1693742304 [2023-09-07 16:48:15,982 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105292, schain_finish_ts: 1693742004 [2023-09-07 16:48:18,283 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:48:18,284 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:48:18,287 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:48:18,287 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:48:18,288 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:48:18,291 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:48:18,295 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:48:18,296 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 16:48:18,327 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:48:18,595 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:48:18,776 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:48:19,007 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:48:19,009 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2478 [2023-09-07 16:48:19,011 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:48:19,011 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:48:19,011 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:48:19,012 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:48:19,012 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:48:19,018 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:48:19,020 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:48:19.020406 [2023-09-07 16:48:19,156 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:48:19,193 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:48:19,232 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:48:19,235 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:48:19,244 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:48:19,245 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694105299.1908085, 'te': 1694105299.1994333, 'initial_status': True} ====================================================================================================  [2023-09-07 16:48:19,249 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:48:19.249635 [2023-09-07 16:48:19,254 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:48:19,255 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:48:23,488 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105292, schain_finish_ts: 0 [2023-09-07 16:48:27,104 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105292, schain_finish_ts: 1692377364 [2023-09-07 16:48:27,182 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:48:27,183 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:48:27,183 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:48:27,183 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:48:27,184 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:48:27,184 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 16:48:27,210 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:48:27,277 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:48:27,277 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:48:27,278 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:48:27,278 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:48:27,278 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:48:27,279 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:48:27,283 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:48:27,285 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 16:48:27,311 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:48:27,464 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:48:27,610 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:48:27,769 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:48:27,921 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:48:27,922 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2483 [2023-09-07 16:48:27,923 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:48:27,923 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:48:27,923 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:48:27,924 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:48:27,924 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:48:27,930 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:48:27,932 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:48:27.932684 [2023-09-07 16:48:28,054 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:48:28,166 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:48:28,167 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2476 [2023-09-07 16:48:28,168 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:48:28,168 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:48:28,169 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:48:28,169 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:48:28,170 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:48:28,176 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:48:28,179 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:48:28.179237 [2023-09-07 16:48:28,195 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:48:28,221 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:48:28,249 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:48:28,252 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:48:28,259 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:48:28,259 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694105308.2196696, 'te': 1694105308.2253754, 'initial_status': True} ====================================================================================================  [2023-09-07 16:48:28,262 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:48:28.262149 [2023-09-07 16:48:28,266 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:48:28,267 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:48:28,331 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:48:28,364 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:48:28,394 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:48:28,396 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:48:28,404 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:48:28,404 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694105308.3625567, 'te': 1694105308.3687112, 'initial_status': True} ====================================================================================================  [2023-09-07 16:48:28,406 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:48:28.406639 [2023-09-07 16:48:28,410 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:48:28,411 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:48:39,122 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:48:39,122 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:48:39,123 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 16:48:40,196 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:48:40,196 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:48:40,196 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:48:40,197 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:48:40,197 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:48:40,198 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 16:48:40,225 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:48:40,472 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:48:40,609 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:48:40,745 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:48:40,750 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2476 [2023-09-07 16:48:40,756 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:48:40,757 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:48:40,759 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:48:40,762 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:48:40,765 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:48:40,782 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:48:40,786 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:48:40.786413 [2023-09-07 16:48:41,156 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:48:41,186 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:48:41,218 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:48:41,220 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:48:41,227 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:48:41,228 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694105321.1840816, 'te': 1694105321.1910698, 'initial_status': True} ====================================================================================================  [2023-09-07 16:48:41,229 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:48:41.229775 [2023-09-07 16:48:41,233 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:48:41,234 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:48:52,575 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:48:52,576 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:48:52,576 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:48:52,576 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:48:52,576 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:48:52,577 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:48:52,580 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:48:52,587 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 16:48:52,631 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:48:52,933 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:48:53,320 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:48:53,452 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:48:53,453 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2461 [2023-09-07 16:48:53,454 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:48:53,455 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:48:53,455 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:48:53,455 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:48:53,456 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:48:53,467 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:48:53,469 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:48:53.469655 [2023-09-07 16:48:53,632 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:48:53,661 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:48:53,705 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:48:53,707 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:48:53,715 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:48:53,715 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694105333.6590018, 'te': 1694105333.6658852, 'initial_status': True} ====================================================================================================  [2023-09-07 16:48:53,717 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:48:53.717269 [2023-09-07 16:48:53,723 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:48:53,724 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:48:54,169 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:48:54,169 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:48:54,171 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:48:54.171499 [2023-09-07 16:48:54,178 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:48:54,178 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:48:54,184 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:48:54,188 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694105334.176612, 'te': 1694105334.187886, 'initial_status': True} ====================================================================================================  [2023-09-07 16:48:54,189 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:48:54.189770 [2023-09-07 16:48:54,194 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:48:54,194 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:49:00,931 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105328, schain_finish_ts: 1693412520 [2023-09-07 16:49:05,101 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105328, schain_finish_ts: 1693742304 [2023-09-07 16:49:05,540 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105328, schain_finish_ts: 1692637668 [2023-09-07 16:49:05,664 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:49:05,664 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:49:05,666 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:49:05.666623 [2023-09-07 16:49:05,674 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:49:05,674 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:49:05,680 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:49:05,684 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694105345.6728501, 'te': 1694105345.6847048, 'initial_status': True} ====================================================================================================  [2023-09-07 16:49:05,686 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:49:05.686462 [2023-09-07 16:49:05,691 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:49:05,691 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:49:08,095 INFO][1][MainProcess][MainThread] - core.schains.process_manager:57 - Process manager started [2023-09-07 16:49:12,726 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105328, schain_finish_ts: 1693742004 [2023-09-07 16:49:13,546 INFO][1][MainProcess][MainThread] - core.schains.notifications:35 - Node account has 406869999689179237 WEI [2023-09-07 16:49:13,548 INFO][1][MainProcess][MainThread] - tools.notifications.messages:188 - Saving new balance state 3006 1 [2023-09-07 16:49:13,555 INFO][1][MainProcess][MainThread] - core.schains.process_manager:95 - Fetching schains to monitor... [2023-09-07 16:49:17,633 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:49:17,633 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:49:17,633 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:49:17,634 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:49:17,634 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:49:17,635 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 16:49:17,661 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:49:17,821 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:49:17,821 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:49:17,821 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:49:17,821 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:49:17,821 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:49:17,821 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:49:17,823 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:49:17,824 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 16:49:17,874 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:49:17,920 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:49:18,182 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:49:18,340 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:49:18,379 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:49:18,381 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2484 [2023-09-07 16:49:18,382 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:49:18,383 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:49:18,383 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:49:18,384 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:49:18,384 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:49:18,397 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:49:18,401 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:49:18.401700 [2023-09-07 16:49:18,533 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:49:18,533 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:49:18,540 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:49:18.540056 [2023-09-07 16:49:18,543 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:49:18,549 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:49:18,549 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:49:18,559 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:49:18,566 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694105358.5481954, 'te': 1694105358.5657423, 'initial_status': True} ====================================================================================================  [2023-09-07 16:49:18,570 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:49:18.570830 [2023-09-07 16:49:18,576 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:49:18,577 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:49:18,692 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:49:18,693 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2479 [2023-09-07 16:49:18,694 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:49:18,694 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:49:18,695 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:49:18,695 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:49:18,695 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:49:18,701 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:49:18,703 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:49:18.703670 [2023-09-07 16:49:18,732 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:49:18,772 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:49:18,845 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:49:18,848 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:49:18,871 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:49:18,872 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694105358.7700062, 'te': 1694105358.778945, 'initial_status': True} ====================================================================================================  [2023-09-07 16:49:18,874 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:49:18.874397 [2023-09-07 16:49:18,880 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:49:18,881 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:49:18,972 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:49:19,001 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:49:19,031 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:49:19,033 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:49:19,041 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:49:19,041 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694105358.9988432, 'te': 1694105359.0070498, 'initial_status': True} ====================================================================================================  [2023-09-07 16:49:19,043 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:49:19.043538 [2023-09-07 16:49:19,049 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:49:19,050 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:49:27,930 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:49:27,931 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:49:27,931 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:49:27,931 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:49:27,932 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:49:27,933 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 16:49:27,954 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:49:28,029 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105364, schain_finish_ts: 1692377364 [2023-09-07 16:49:28,165 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:49:28,297 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:49:28,396 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:49:28,397 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2477 [2023-09-07 16:49:28,398 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:49:28,398 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:49:28,399 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:49:28,399 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:49:28,399 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:49:28,405 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:49:28,407 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:49:28.407694 [2023-09-07 16:49:28,554 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:49:28,581 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:49:28,609 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:49:28,611 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:49:28,618 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:49:28,618 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694105368.5796473, 'te': 1694105368.5854702, 'initial_status': True} ====================================================================================================  [2023-09-07 16:49:28,620 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:49:28.620168 [2023-09-07 16:49:28,624 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:49:28,625 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:49:30,133 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:49:30,134 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:49:30,135 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:49:30,135 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:49:30,136 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:49:30,137 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:49:30,143 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:49:30,144 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 16:49:30,164 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:49:30,424 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:49:30,581 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:49:30,725 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:49:30,726 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2477 [2023-09-07 16:49:30,727 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:49:30,727 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:49:30,727 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:49:30,727 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:49:30,728 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:49:30,735 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:49:30,737 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:49:30.737141 [2023-09-07 16:49:30,908 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:49:30,937 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:49:30,965 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:49:30,967 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:49:30,975 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:49:30,975 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694105370.9357557, 'te': 1694105370.9423358, 'initial_status': True} ====================================================================================================  [2023-09-07 16:49:30,977 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:49:30.977300 [2023-09-07 16:49:30,982 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:49:30,982 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:49:37,704 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105376, schain_finish_ts: 0 [2023-09-07 16:49:37,863 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:49:37,863 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:49:37,865 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:49:37.865410 [2023-09-07 16:49:37,873 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:49:37,873 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:49:37,880 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:49:37,886 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694105377.87073, 'te': 1694105377.8858106, 'initial_status': True} ====================================================================================================  [2023-09-07 16:49:37,889 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:49:37.889647 [2023-09-07 16:49:37,894 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:49:37,894 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:49:49,172 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105376, schain_finish_ts: 1693412520 [2023-09-07 16:49:51,019 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:49:51,020 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:49:51,020 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:49:51,020 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:49:51,021 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:49:51,021 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:49:51,023 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:49:51,025 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 16:49:51,057 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:49:51,329 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:49:51,529 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:49:51,659 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:49:51,660 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2462 [2023-09-07 16:49:51,661 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:49:51,661 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:49:51,662 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:49:51,662 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:49:51,662 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:49:51,669 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:49:51,671 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:49:51.671222 [2023-09-07 16:49:51,810 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:49:51,851 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:49:51,880 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:49:51,882 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:49:51,889 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:49:51,890 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694105391.8484728, 'te': 1694105391.8557746, 'initial_status': True} ====================================================================================================  [2023-09-07 16:49:51,891 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:49:51.891934 [2023-09-07 16:49:51,896 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:49:51,897 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:49:55,411 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:49:55,414 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:49:55,417 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 16:49:57,079 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105388, schain_finish_ts: 1693742304 [2023-09-07 16:50:01,112 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105388, schain_finish_ts: 1693742004 [2023-09-07 16:50:05,496 INFO][1][MainProcess][MainThread] - core.schains.process_manager:109 - Get leaving_history for node ... [2023-09-07 16:50:05,872 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:50:05,873 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:50:05,875 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:50:05.875716 [2023-09-07 16:50:05,884 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:50:05,884 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:50:05,890 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:50:05,894 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694105405.8825383, 'te': 1694105405.8944812, 'initial_status': True} ====================================================================================================  [2023-09-07 16:50:05,896 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:50:05.896759 [2023-09-07 16:50:05,901 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:50:05,901 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:50:06,890 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:50:06,890 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:50:06,891 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:50:06,891 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:50:06,891 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:50:06,894 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:50:06,896 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:50:06,902 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 16:50:06,926 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:50:07,123 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:50:07,403 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:50:07,507 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:50:07,507 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2480 [2023-09-07 16:50:07,508 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:50:07,508 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:50:07,509 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:50:07,509 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:50:07,509 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:50:07,516 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:50:07,518 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:50:07.518222 [2023-09-07 16:50:07,672 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:50:07,701 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:50:07,737 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:50:07,740 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:50:07,749 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:50:07,749 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694105407.6990352, 'te': 1694105407.7053618, 'initial_status': True} ====================================================================================================  [2023-09-07 16:50:07,752 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:50:07.752334 [2023-09-07 16:50:07,756 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:50:07,757 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:50:09,328 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:50:09,328 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:50:09,328 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:50:09,328 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:50:09,328 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:50:09,329 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 16:50:09,349 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:50:09,559 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:50:09,715 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:50:09,873 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:50:09,874 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2478 [2023-09-07 16:50:09,875 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:50:09,876 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:50:09,877 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:50:09,877 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:50:09,878 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:50:09,888 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:50:09,892 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:50:09.892352 [2023-09-07 16:50:10,445 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:50:10,475 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:50:10,511 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:50:10,515 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:50:10,524 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:50:10,525 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694105410.4733694, 'te': 1694105410.479971, 'initial_status': True} ====================================================================================================  [2023-09-07 16:50:10,528 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:50:10.528092 [2023-09-07 16:50:10,532 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:50:10,533 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:50:18,893 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:50:18,894 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:50:18,896 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:50:18.896043 [2023-09-07 16:50:18,903 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:50:18,903 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:50:18,908 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:50:18,913 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694105418.9010518, 'te': 1694105418.9130008, 'initial_status': True} ====================================================================================================  [2023-09-07 16:50:18,915 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:50:18.914993 [2023-09-07 16:50:18,920 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:50:18,920 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:50:19,950 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:50:19,950 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:50:19,950 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:50:19,951 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:50:19,951 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:50:19,951 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 16:50:19,970 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:50:20,159 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:50:20,292 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:50:20,458 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:50:20,459 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2485 [2023-09-07 16:50:20,459 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:50:20,460 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:50:20,460 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:50:20,460 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:50:20,460 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:50:20,466 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:50:20,468 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:50:20.468533 [2023-09-07 16:50:20,706 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:50:20,730 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:50:20,757 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:50:20,759 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:50:20,765 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:50:20,765 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694105420.7290936, 'te': 1694105420.7356293, 'initial_status': True} ====================================================================================================  [2023-09-07 16:50:20,767 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:50:20.767233 [2023-09-07 16:50:20,770 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:50:20,771 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:50:24,338 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105412, schain_finish_ts: 1692637668 [2023-09-07 16:50:29,910 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105424, schain_finish_ts: 0 [2023-09-07 16:50:31,694 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105424, schain_finish_ts: 1692377616 [2023-09-07 16:50:35,505 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:50:35,505 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:50:35,507 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:50:35.507277 [2023-09-07 16:50:35,513 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:50:35,514 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:50:35,519 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:50:35,522 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694105435.5124974, 'te': 1694105435.5227096, 'initial_status': True} ====================================================================================================  [2023-09-07 16:50:35,524 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:50:35.524580 [2023-09-07 16:50:35,528 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:50:35,528 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:50:41,547 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105436, schain_finish_ts: 1692377364 [2023-09-07 16:50:41,839 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105436, schain_finish_ts: 1693412520 [2023-09-07 16:50:46,939 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:50:46,939 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:50:46,940 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:50:46,940 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:50:46,940 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:50:46,941 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:50:46,945 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:50:46,946 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 16:50:46,978 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:50:47,235 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:50:47,235 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:50:47,236 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:50:47,236 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:50:47,236 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:50:47,236 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:50:47,240 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:50:47,243 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 16:50:47,277 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:50:47,337 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:50:47,559 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:50:47,726 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:50:47,859 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:50:47,860 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2478 [2023-09-07 16:50:47,861 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:50:47,861 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:50:47,861 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:50:47,862 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:50:47,862 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:50:47,868 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:50:47,871 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:50:47,874 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:50:47.874193 [2023-09-07 16:50:48,012 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:50:48,013 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2463 [2023-09-07 16:50:48,014 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:50:48,014 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:50:48,014 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:50:48,014 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:50:48,015 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:50:48,022 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:50:48,024 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:50:48.024248 [2023-09-07 16:50:48,057 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:50:48,093 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:50:48,144 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:50:48,147 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:50:48,156 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:50:48,157 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694105448.090552, 'te': 1694105448.0997021, 'initial_status': True} ====================================================================================================  [2023-09-07 16:50:48,159 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:50:48.159302 [2023-09-07 16:50:48,169 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:50:48,170 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:50:48,226 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:50:48,258 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:50:48,298 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:50:48,303 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:50:48,310 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:50:48,311 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694105448.2567925, 'te': 1694105448.2644408, 'initial_status': True} ====================================================================================================  [2023-09-07 16:50:48,312 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:50:48.312636 [2023-09-07 16:50:48,318 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:50:48,319 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:50:49,578 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:50:49,579 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:50:49,579 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 16:51:04,338 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:51:04,338 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:51:04,338 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:51:04,339 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:51:04,339 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:51:04,339 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:51:04,341 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:51:04,342 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 16:51:04,362 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:51:04,556 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:51:04,697 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:51:04,796 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:51:04,797 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2481 [2023-09-07 16:51:04,797 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:51:04,797 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:51:04,798 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:51:04,798 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:51:04,798 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:51:04,804 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:51:04,806 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:51:04.806924 [2023-09-07 16:51:05,014 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:51:05,040 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:51:05,069 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:51:05,071 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:51:05,078 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:51:05,078 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694105465.037751, 'te': 1694105465.0452945, 'initial_status': True} ====================================================================================================  [2023-09-07 16:51:05,080 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:51:05.080242 [2023-09-07 16:51:05,084 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:51:05,085 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:51:12,170 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105460, schain_finish_ts: 0 [2023-09-07 16:51:13,627 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105472, schain_finish_ts: 1693742004 [2023-09-07 16:51:15,925 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105472, schain_finish_ts: 1692637668 [2023-09-07 16:51:16,006 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:51:16,007 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:51:16,009 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:51:16.009091 [2023-09-07 16:51:16,015 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:51:16,015 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:51:16,022 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:51:16,027 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694105476.0141075, 'te': 1694105476.0268059, 'initial_status': True} ====================================================================================================  [2023-09-07 16:51:16,029 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:51:16.029537 [2023-09-07 16:51:16,035 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:51:16,035 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:51:16,742 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105472, schain_finish_ts: 1693742304 [2023-09-07 16:51:18,905 INFO][1][MainProcess][MainThread] - core.schains.process_manager:117 - Got leaving sChains for the node: [] [2023-09-07 16:51:18,905 INFO][1][MainProcess][MainThread] - core.schains.process_manager:101 -  ==================================================================================================== Monitoring sChains Node ID: 16 sChains on node: [{'name': 'portly-passionate-sirius', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666281384, 'startBlock': 7803380, 'deposit': 0, 'index': 1, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x0000017723ca2', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'curly-red-alterf', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1676058936, 'startBlock': 8469646, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000001fe6cdfa', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'howling-hoedus', 'mainnetOwner': '0x1026467C91fdA40aCc7e6935d1E52f67274FD669', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666181400, 'startBlock': 7796660, 'deposit': 0, 'index': 0, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0xe5a600f4c2608', 'multitransactionMode': False, 'thresholdEncryption': False, 'active': True}, {'name': 'green-giddy-denebola', 'mainnetOwner': '0x8e835d5d9E28ffD259E8Fa8988Cacfc333A582B4', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674565164, 'startBlock': 8368049, 'deposit': 0, 'index': 4, 'generation': 1, 'originator': '0xd2001DAb6898127Be2F167B548691C87251D13C3', 'chainId': '0x00000585eb4b1', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'wan-tasty-kitalpha', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674568392, 'startBlock': 8368271, 'deposit': 0, 'index': 5, 'generation': 1, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x00000152e7be9', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'submissive-well-off-rotanev', 'mainnetOwner': '0x487360568fE5Ee4AE5c75782D5F63e9935B0E512', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1692361104, 'startBlock': 9539053, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000003791fa10', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}] Number of sChains on node: 6 Empty sChain structs: 1 ====================================================================================================  [2023-09-07 16:51:21,740 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:51:21,740 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:51:21,741 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:51:21,741 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:51:21,741 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:51:21,741 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 16:51:21,765 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:51:21,956 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:51:22,124 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:51:22,252 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:51:22,253 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2479 [2023-09-07 16:51:22,254 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:51:22,254 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:51:22,254 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:51:22,255 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:51:22,255 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:51:22,266 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:51:22,268 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:51:22.268670 [2023-09-07 16:51:22,357 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:51:22,357 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain portly-passionate-sirius - Process is running: PID = 7 [2023-09-07 16:51:22,432 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:51:22,466 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:51:22,500 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:51:22,503 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:51:22,510 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:51:22,510 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694105482.4647424, 'te': 1694105482.471614, 'initial_status': True} ====================================================================================================  [2023-09-07 16:51:22,512 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:51:22.512305 [2023-09-07 16:51:22,516 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:51:22,517 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:51:25,561 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:51:25,562 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain curly-red-alterf - Process is running: PID = 8 [2023-09-07 16:51:27,366 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:51:27,366 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:51:27,370 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:51:27.370601 [2023-09-07 16:51:27,382 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:51:27,383 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:51:27,389 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:51:27,394 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694105487.3782444, 'te': 1694105487.3938367, 'initial_status': True} ====================================================================================================  [2023-09-07 16:51:27,395 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:51:27.395839 [2023-09-07 16:51:27,400 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:51:27,400 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:51:30,266 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:51:30,266 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain howling-hoedus - Process is running: PID = 9 [2023-09-07 16:51:30,337 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:51:30,337 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:51:30,337 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:51:30,338 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:51:30,338 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:51:30,339 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 16:51:30,384 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:51:30,449 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105484, schain_finish_ts: 1692377364 [2023-09-07 16:51:30,666 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:51:30,871 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:51:30,889 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:51:30,889 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:51:30,894 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:51:30.894904 [2023-09-07 16:51:30,906 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:51:30,906 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:51:30,914 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:51:30,919 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694105490.904494, 'te': 1694105490.9189882, 'initial_status': True} ====================================================================================================  [2023-09-07 16:51:30,920 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:51:30.920901 [2023-09-07 16:51:30,925 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:51:30,925 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:51:31,067 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:51:31,069 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2486 [2023-09-07 16:51:31,071 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:51:31,072 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:51:31,072 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:51:31,072 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:51:31,073 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:51:31,091 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:51:31,097 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:51:31.097354 [2023-09-07 16:51:31,253 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:51:31,287 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:51:31,318 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:51:31,321 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:51:31,329 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:51:31,329 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694105491.2851236, 'te': 1694105491.2919562, 'initial_status': True} ====================================================================================================  [2023-09-07 16:51:31,331 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:51:31.331515 [2023-09-07 16:51:31,336 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:51:31,336 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:51:34,123 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:51:34,123 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain green-giddy-denebola - Process is running: PID = 10 [2023-09-07 16:51:35,220 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 16:51:35,220 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 16:51:35,546 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:51:35,546 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain wan-tasty-kitalpha - Process is running: PID = 11 [2023-09-07 16:51:35,588 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:51:35,589 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:51:35,589 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:51:35,589 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:51:35,590 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:51:35,590 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:51:35,591 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:51:35,594 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 16:51:35,625 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:51:35,863 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:51:36,074 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:51:36,276 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:51:36,278 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2479 [2023-09-07 16:51:36,278 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:51:36,279 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:51:36,279 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:51:36,280 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:51:36,280 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:51:36,287 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:51:36,289 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:51:36.289501 [2023-09-07 16:51:36,450 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:51:36,450 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:51:36,451 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:51:36,451 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:51:36,451 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:51:36,451 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:51:36,453 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:51:36,454 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:51:36,455 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 16:51:36,489 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:51:36,490 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:51:36,537 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:51:36,540 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:51:36,555 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:51:36,555 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694105496.4837139, 'te': 1694105496.4958272, 'initial_status': True} ====================================================================================================  [2023-09-07 16:51:36,557 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:51:36.557612 [2023-09-07 16:51:36,562 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:51:36,563 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:51:36,683 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:51:36,683 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain submissive-well-off-rotanev - Process is running: PID = 24912 [2023-09-07 16:51:36,684 INFO][1][MainProcess][MainThread] - core.schains.process_manager:88 - Process manager procedure finished [2023-09-07 16:51:36,684 INFO][1][MainProcess][MainThread] - __main__:65 - Sleeping for 90s after run_process_manager [2023-09-07 16:51:36,776 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:51:36,934 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:51:37,039 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:51:37,040 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2464 [2023-09-07 16:51:37,041 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:51:37,041 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:51:37,041 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:51:37,042 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:51:37,042 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:51:37,049 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:51:37,051 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:51:37.051263 [2023-09-07 16:51:37,213 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:51:37,239 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:51:37,272 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:51:37,274 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:51:37,282 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:51:37,282 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694105497.2377856, 'te': 1694105497.2447674, 'initial_status': True} ====================================================================================================  [2023-09-07 16:51:37,283 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:51:37.283894 [2023-09-07 16:51:37,288 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:51:37,288 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:51:39,747 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105496, schain_finish_ts: 1693412520 [2023-09-07 16:51:42,306 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105496, schain_finish_ts: 0 [2023-09-07 16:51:46,864 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:51:46,864 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:51:46,868 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:51:46.868700 [2023-09-07 16:51:46,876 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:51:46,877 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:51:46,886 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:51:46,894 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694105506.874548, 'te': 1694105506.8934536, 'initial_status': True} ====================================================================================================  [2023-09-07 16:51:46,898 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:51:46.898521 [2023-09-07 16:51:46,905 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:51:46,905 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:51:48,377 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:51:48,377 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:51:48,378 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:51:48,378 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:51:48,378 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:51:48,379 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:51:48,379 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:51:48,380 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 16:51:48,418 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:51:48,785 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:51:48,913 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:51:49,023 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:51:49,025 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2482 [2023-09-07 16:51:49,026 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:51:49,027 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:51:49,029 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:51:49,030 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:51:49,031 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:51:49,045 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:51:49,053 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:51:49.053254 [2023-09-07 16:51:49,358 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:51:49,390 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:51:49,426 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:51:49,428 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:51:49,435 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:51:49,436 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694105509.3883867, 'te': 1694105509.3966918, 'initial_status': True} ====================================================================================================  [2023-09-07 16:51:49,437 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:51:49.437963 [2023-09-07 16:51:49,442 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:51:49,443 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:51:49,878 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:51:49,878 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:51:49,881 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:51:49.881202 [2023-09-07 16:51:49,897 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:51:49,897 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:51:49,904 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:51:49,914 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694105509.8899133, 'te': 1694105509.9147396, 'initial_status': True} ====================================================================================================  [2023-09-07 16:51:49,917 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:51:49.917711 [2023-09-07 16:51:49,922 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:51:49,923 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:51:51,480 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:51:51,480 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:51:51,480 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 16:51:55,507 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 16:51:55,687 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 16:51:55,687 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 16:51:55,688 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 16:51:55,689 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 16:51:55,867 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 16:51:55,868 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 16:51:55,872 INFO][8][curly-red-alterf][T_0] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 16:51:55,878 ERROR][8][curly-red-alterf][T_0] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 16:51:55,879 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:52:01,875 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:40 - Collecting rotation history for submissive-well-off-rotanev... [2023-09-07 16:52:11,050 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105508, schain_finish_ts: 1693742304 [2023-09-07 16:52:11,058 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105508, schain_finish_ts: 1692637668 [2023-09-07 16:52:12,369 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105508, schain_finish_ts: 1693742004 [2023-09-07 16:52:17,356 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105532, schain_finish_ts: 1692377364 [2023-09-07 16:52:18,349 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:52:18,350 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:52:18,350 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:52:18,350 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:52:18,350 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:52:18,351 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 16:52:18,382 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:52:18,649 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:52:18,835 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:50 - Rotation data for submissive-well-off-rotanev: Rotation(leaving_node_id=65, new_node_id=65, freeze_until=1694108100, rotation_counter=8) [2023-09-07 16:52:18,862 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:52:18,967 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:52:18,968 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2487 [2023-09-07 16:52:18,968 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:52:18,969 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:52:18,969 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:52:18,969 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:52:18,969 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:52:18,975 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:52:18,977 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:52:18.977551 [2023-09-07 16:52:19,230 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:52:19,262 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:52:19,288 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:52:19,288 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:52:19,291 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:52:19.291250 [2023-09-07 16:52:19,300 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:52:19,300 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:52:19,301 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:52:19,309 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:52:19,315 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694105539.298041, 'te': 1694105539.3156574, 'initial_status': True} ====================================================================================================  [2023-09-07 16:52:19,318 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:52:19.318219 [2023-09-07 16:52:19,323 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:52:19,324 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:52:19,338 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:52:19,345 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:52:19,345 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694105539.2604783, 'te': 1694105539.2684817, 'initial_status': True} ====================================================================================================  [2023-09-07 16:52:19,347 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:52:19.347266 [2023-09-07 16:52:19,351 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:52:19,352 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:52:22,806 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:52:22,807 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:52:22,807 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:52:22,807 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 16:52:22,808 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 16:52:22,808 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:52:22,812 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:52:22,815 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 16:52:22,864 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:52:23,131 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:52:23,397 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:52:23,518 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:52:23,519 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2480 [2023-09-07 16:52:23,520 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:52:23,521 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:52:23,521 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:52:23,521 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:52:23,522 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:52:23,527 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:52:23,527 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:52:23,527 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:52:23,528 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:52:23,528 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:52:23,529 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:52:23,528 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:52:23,532 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:52:23.532496 [2023-09-07 16:52:23,534 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:52:23,536 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 16:52:23,569 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:52:23,783 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:52:23,814 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:52:23,867 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:52:23,869 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:52:23,878 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:52:23,878 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694105543.8109472, 'te': 1694105543.8198085, 'initial_status': True} ====================================================================================================  [2023-09-07 16:52:23,882 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:52:23.882256 [2023-09-07 16:52:23,886 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:52:23,892 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:52:24,072 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:52:24,255 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:52:24,408 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:52:24,409 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2480 [2023-09-07 16:52:24,410 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:52:24,410 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:52:24,411 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:52:24,411 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:52:24,411 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:52:24,417 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:52:24,419 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:52:24.419626 [2023-09-07 16:52:24,564 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:52:24,609 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:52:24,646 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:52:24,648 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:52:24,656 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:52:24,656 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694105544.605138, 'te': 1694105544.6183896, 'initial_status': True} ====================================================================================================  [2023-09-07 16:52:24,658 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:52:24.658678 [2023-09-07 16:52:24,663 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:52:24,663 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:52:30,047 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:52:30,048 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:52:30,050 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:52:30.050623 [2023-09-07 16:52:30,058 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:52:30,059 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:52:30,064 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:52:30,069 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694105550.0567741, 'te': 1694105550.0688787, 'initial_status': True} ====================================================================================================  [2023-09-07 16:52:30,070 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:52:30.070820 [2023-09-07 16:52:30,074 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:52:30,075 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:52:31,033 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:52:31,033 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:52:31,033 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:52:31,033 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:52:31,034 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:52:31,034 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:52:31,036 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:52:31,037 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 16:52:31,081 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:52:31,255 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105544, schain_finish_ts: 0 [2023-09-07 16:52:31,472 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:52:31,633 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:52:31,757 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:52:31,760 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2465 [2023-09-07 16:52:31,761 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:52:31,761 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:52:31,762 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:52:31,763 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:52:31,763 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:52:31,769 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:52:31,771 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:52:31.771720 [2023-09-07 16:52:31,910 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:52:31,937 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:52:31,971 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:52:31,973 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:52:31,981 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:52:31,981 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694105551.9356692, 'te': 1694105551.9427693, 'initial_status': True} ====================================================================================================  [2023-09-07 16:52:31,983 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:52:31.983139 [2023-09-07 16:52:31,987 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:52:31,988 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:52:33,936 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 16:52:33,937 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:52:33,939 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:52:33.939585 [2023-09-07 16:52:33,945 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 16:52:35,065 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105544, schain_finish_ts: 1693412520 [2023-09-07 16:52:39,079 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:52:39,079 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:52:39,079 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 16:52:44,214 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:52:44,214 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:52:44,214 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:52:44,214 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:52:44,215 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:52:44,215 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:52:44,217 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:52:44,219 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 16:52:44,244 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:52:44,525 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:52:44,706 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:52:44,807 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:52:44,808 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2483 [2023-09-07 16:52:44,809 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:52:44,809 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:52:44,809 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:52:44,809 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:52:44,810 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:52:44,815 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:52:44,817 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:52:44.817561 [2023-09-07 16:52:45,043 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:52:45,069 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:52:45,082 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:52:45,082 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:52:45,084 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:52:45.084898 [2023-09-07 16:52:45,092 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:52:45,092 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:52:45,098 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:52:45,102 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694105565.0906188, 'te': 1694105565.1025279, 'initial_status': True} ====================================================================================================  [2023-09-07 16:52:45,103 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:52:45,104 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:52:45.104410 [2023-09-07 16:52:45,108 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:52:45,108 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:52:45,108 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:52:45,116 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:52:45,116 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694105565.0677161, 'te': 1694105565.074807, 'initial_status': True} ====================================================================================================  [2023-09-07 16:52:45,118 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:52:45.118212 [2023-09-07 16:52:45,122 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:52:45,123 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:52:47,562 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105556, schain_finish_ts: 1693742304 [2023-09-07 16:52:49,539 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105556, schain_finish_ts: 1692637668 [2023-09-07 16:52:51,138 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:52:51,139 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:52:51,141 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:52:51.141055 [2023-09-07 16:52:51,148 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:52:51,148 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:52:51,154 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:52:51,159 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694105571.1463938, 'te': 1694105571.1593735, 'initial_status': True} ====================================================================================================  [2023-09-07 16:52:51,161 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:52:51.161177 [2023-09-07 16:52:51,165 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:52:51,165 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:52:51,233 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:52:51,233 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:52:51,233 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:52:51,234 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:52:51,234 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:52:51,234 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 16:52:51,263 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:52:51,540 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:52:51,703 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:52:51,809 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:52:51,810 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2488 [2023-09-07 16:52:51,810 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:52:51,811 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:52:51,811 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:52:51,812 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:52:51,812 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:52:51,819 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:52:51,821 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:52:51.821290 [2023-09-07 16:52:52,004 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:52:52,037 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:52:52,078 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:52:52,080 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:52:52,088 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:52:52,089 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694105572.0352244, 'te': 1694105572.041932, 'initial_status': True} ====================================================================================================  [2023-09-07 16:52:52,091 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:52:52.091202 [2023-09-07 16:52:52,095 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:52:52,096 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:53:00,646 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:53:00,646 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:53:00,646 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:53:00,647 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:53:00,648 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:53:00,648 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:53:00,649 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:53:00,652 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 16:53:00,677 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:53:00,846 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:53:01,008 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:53:01,119 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:53:01,120 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2481 [2023-09-07 16:53:01,121 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:53:01,122 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:53:01,122 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:53:01,122 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:53:01,123 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:53:01,132 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:53:01,134 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:53:01.134622 [2023-09-07 16:53:01,287 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:53:01,323 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:53:01,363 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:53:01,366 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:53:01,374 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:53:01,374 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694105581.3151593, 'te': 1694105581.3290246, 'initial_status': True} ====================================================================================================  [2023-09-07 16:53:01,376 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:53:01.376422 [2023-09-07 16:53:01,381 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:53:01,381 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:53:01,737 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105568, schain_finish_ts: 1693742004 [2023-09-07 16:53:04,807 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105568, schain_finish_ts: 1692377364 [2023-09-07 16:53:06,743 INFO][1][MainProcess][MainThread] - core.schains.cleaner:65 - Cleaner process started [2023-09-07 16:53:06,777 INFO][30512][cleaner][MainThread] - core.schains.cleaner:110 - Cleaner procedure started. [2023-09-07 16:53:17,135 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:53:17,136 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:53:17,136 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:53:17,136 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:53:17,136 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:53:17,137 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:53:17,139 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:53:17,140 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 16:53:17,167 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:53:17,391 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:53:17,584 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:53:17,721 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:53:17,722 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2466 [2023-09-07 16:53:17,723 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:53:17,723 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:53:17,723 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:53:17,724 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:53:17,724 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:53:17,732 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:53:17,736 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:53:17.735990 [2023-09-07 16:53:17,897 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:53:17,922 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:53:17,954 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:53:17,957 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:53:17,966 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:53:17,967 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694105597.921266, 'te': 1694105597.9281774, 'initial_status': True} ====================================================================================================  [2023-09-07 16:53:17,970 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:53:17.969986 [2023-09-07 16:53:17,975 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:53:17,976 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:53:18,523 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:53:18,525 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:53:18,525 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:53:18,525 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:53:18,525 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:53:18,526 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 16:53:18,546 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:53:18,605 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105592, schain_finish_ts: 1693412520 [2023-09-07 16:53:18,769 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:53:18,919 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:53:19,029 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:53:19,030 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2481 [2023-09-07 16:53:19,031 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:53:19,031 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:53:19,032 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:53:19,032 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:53:19,032 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:53:19,041 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:53:19,043 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:53:19.043187 [2023-09-07 16:53:19,202 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:53:19,230 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:53:19,284 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:53:19,286 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:53:19,296 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:53:19,297 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694105599.228957, 'te': 1694105599.2374806, 'initial_status': True} ====================================================================================================  [2023-09-07 16:53:19,299 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:53:19.299943 [2023-09-07 16:53:19,305 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:53:19,306 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:53:26,057 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105592, schain_finish_ts: 1693742304 [2023-09-07 16:53:32,262 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:53:32,263 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:53:32,265 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:53:32,266 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:53:32,267 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:53:32,267 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:53:32,269 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:53:32,273 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 16:53:32,297 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:53:32,478 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:53:32,523 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105604, schain_finish_ts: 0 [2023-09-07 16:53:32,624 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:53:32,730 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:53:32,731 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2484 [2023-09-07 16:53:32,731 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:53:32,732 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:53:32,732 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:53:32,732 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:53:32,733 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:53:32,739 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:53:32,741 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:53:32.741715 [2023-09-07 16:53:32,902 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:53:32,933 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:53:32,965 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:53:32,967 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:53:32,976 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:53:32,976 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694105612.9315107, 'te': 1694105612.9388146, 'initial_status': True} ====================================================================================================  [2023-09-07 16:53:32,978 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:53:32.977999 [2023-09-07 16:53:32,982 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:53:32,985 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:53:33,581 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:53:33,581 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:53:33,584 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:53:33.584571 [2023-09-07 16:53:33,593 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:53:33,593 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:53:33,601 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:53:33,606 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694105613.5914433, 'te': 1694105613.606508, 'initial_status': True} ====================================================================================================  [2023-09-07 16:53:33,609 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:53:33.609341 [2023-09-07 16:53:33,615 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:53:33,616 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:53:38,733 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:53:38,733 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:53:38,733 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:53:38,733 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:53:38,733 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:53:38,734 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 16:53:38,754 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:53:38,984 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:53:38,986 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:53:38,989 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 16:53:38,997 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:53:39,172 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:53:39,293 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:53:39,294 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2489 [2023-09-07 16:53:39,294 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:53:39,294 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:53:39,295 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:53:39,295 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:53:39,295 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:53:39,302 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:53:39,304 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:53:39.304532 [2023-09-07 16:53:39,544 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:53:39,574 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:53:39,601 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:53:39,603 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:53:39,610 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:53:39,610 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694105619.5719903, 'te': 1694105619.5786002, 'initial_status': True} ====================================================================================================  [2023-09-07 16:53:39,612 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:53:39.612202 [2023-09-07 16:53:39,616 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:53:39,616 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:53:45,257 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:53:45,257 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:53:45,259 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:53:45.259762 [2023-09-07 16:53:45,267 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:53:45,268 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:53:45,276 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:53:45,281 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694105625.266458, 'te': 1694105625.2812395, 'initial_status': True} ====================================================================================================  [2023-09-07 16:53:45,283 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:53:45.283966 [2023-09-07 16:53:45,289 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:53:45,289 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:53:45,308 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105616, schain_finish_ts: 1692637668 [2023-09-07 16:53:51,692 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:53:51,692 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:53:51,692 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:53:51,692 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:53:51,693 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:53:51,693 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:53:51,694 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:53:51,697 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 16:53:51,722 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:53:52,046 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:53:52,209 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:53:52,376 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:53:52,377 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2482 [2023-09-07 16:53:52,377 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:53:52,377 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:53:52,378 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:53:52,378 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:53:52,379 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:53:52,386 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:53:52,389 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:53:52.389312 [2023-09-07 16:53:52,447 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:53:52,448 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:53:52,451 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:53:52.451045 [2023-09-07 16:53:52,473 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:53:52,473 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:53:52,479 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:53:52,483 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694105632.471693, 'te': 1694105632.4835482, 'initial_status': True} ====================================================================================================  [2023-09-07 16:53:52,485 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:53:52.485423 [2023-09-07 16:53:52,489 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:53:52,490 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:53:52,542 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:53:52,574 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:53:52,586 INFO][30512][cleaner][MainThread] - core.schains.cleaner:116 - sChains on contracts: ['portly-passionate-sirius', 'curly-red-alterf', 'howling-hoedus', 'green-giddy-denebola', 'wan-tasty-kitalpha', '', 'submissive-well-off-rotanev'] sChains on node: ['curly-red-alterf', 'green-giddy-denebola', 'howling-hoedus', 'portly-passionate-sirius', 'submissive-well-off-rotanev', 'wan-tasty-kitalpha'] [2023-09-07 16:53:52,586 INFO][30512][cleaner][MainThread] - core.schains.cleaner:132 - Cleanup procedure finished [2023-09-07 16:53:52,593 INFO][1][MainProcess][MainThread] - core.schains.cleaner:67 - Cleaner process is joined. [2023-09-07 16:53:52,594 INFO][1][MainProcess][MainThread] - core.schains.cleaner:68 - Terminating the process [2023-09-07 16:53:52,594 INFO][1][MainProcess][MainThread] - __main__:70 - Sleeping for 90s after run_cleaner [2023-09-07 16:53:52,604 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105628, schain_finish_ts: 1693742004 [2023-09-07 16:53:52,619 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:53:52,622 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:53:52,631 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:53:52,632 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694105632.5726173, 'te': 1694105632.5813544, 'initial_status': True} ====================================================================================================  [2023-09-07 16:53:52,633 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:53:52.633623 [2023-09-07 16:53:52,638 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:53:52,638 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:53:55,796 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105628, schain_finish_ts: 1692377364 [2023-09-07 16:53:55,988 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:53:55,989 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:53:55,989 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:53:55,989 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:53:55,989 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:53:55,990 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 16:53:56,021 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:53:56,194 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:53:56,194 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:53:56,196 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:53:56.196466 [2023-09-07 16:53:56,203 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:53:56,204 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:53:56,209 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:53:56,213 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694105636.2023475, 'te': 1694105636.2132266, 'initial_status': True} ====================================================================================================  [2023-09-07 16:53:56,215 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:53:56.215126 [2023-09-07 16:53:56,219 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:53:56,219 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:53:56,326 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:53:56,674 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:53:56,797 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:53:56,798 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2482 [2023-09-07 16:53:56,799 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:53:56,799 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:53:56,800 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:53:56,800 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:53:56,800 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:53:56,807 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:53:56,809 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:53:56.809773 [2023-09-07 16:53:56,954 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:53:57,004 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:53:57,031 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:53:57,032 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:53:57,040 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:53:57,040 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694105637.0019643, 'te': 1694105637.0085084, 'initial_status': True} ====================================================================================================  [2023-09-07 16:53:57,042 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:53:57.042178 [2023-09-07 16:53:57,046 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:53:57,046 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:53:59,167 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:53:59,167 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:53:59,167 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:53:59,168 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:53:59,168 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:53:59,168 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:53:59,170 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:53:59,172 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 16:53:59,196 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:53:59,506 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:53:59,648 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:53:59,748 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:53:59,749 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2467 [2023-09-07 16:53:59,749 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:53:59,749 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:53:59,750 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:53:59,750 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:53:59,750 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:53:59,757 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:53:59,758 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:53:59.758922 [2023-09-07 16:53:59,920 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:53:59,953 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:53:59,998 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:54:00,002 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:54:00,028 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:54:00,032 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694105639.951364, 'te': 1694105639.9600115, 'initial_status': True} ====================================================================================================  [2023-09-07 16:54:00,037 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:54:00.037251 [2023-09-07 16:54:00,045 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:54:00,046 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:54:05,973 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:54:05,973 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:54:05,975 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:54:05.975580 [2023-09-07 16:54:05,982 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:54:05,982 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:54:05,988 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:54:05,992 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694105645.9808784, 'te': 1694105645.99226, 'initial_status': True} ====================================================================================================  [2023-09-07 16:54:05,994 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:54:05.994029 [2023-09-07 16:54:05,999 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:54:05,999 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:54:07,987 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105640, schain_finish_ts: 1693742304 [2023-09-07 16:54:08,642 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 63 -> 65 [2023-09-07 16:54:09,126 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:54:09,126 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:54:09,126 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:54:09,126 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:54:09,127 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:54:09,127 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 16:54:09,146 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:54:09,360 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:54:09,507 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:54:09,609 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:54:09,610 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2490 [2023-09-07 16:54:09,610 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:54:09,610 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:54:09,611 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:54:09,611 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:54:09,611 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:54:09,617 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:54:09,619 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:54:09.619953 [2023-09-07 16:54:09,763 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:54:09,791 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:54:09,819 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:54:09,821 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:54:09,828 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:54:09,828 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694105649.789376, 'te': 1694105649.795892, 'initial_status': True} ====================================================================================================  [2023-09-07 16:54:09,829 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:54:09.829802 [2023-09-07 16:54:09,834 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:54:09,835 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:54:13,228 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105652, schain_finish_ts: 1693412520 [2023-09-07 16:54:19,348 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105652, schain_finish_ts: 1692637668 [2023-09-07 16:54:19,827 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:54:19,827 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:54:19,827 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:54:19,827 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:54:19,827 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:54:19,828 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:54:19,831 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:54:19,832 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 16:54:19,860 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:54:20,038 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:54:20,185 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:54:20,200 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 16:54:20,283 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:54:20,284 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2485 [2023-09-07 16:54:20,284 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:54:20,284 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:54:20,285 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:54:20,285 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:54:20,285 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:54:20,291 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:54:20,293 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:54:20.293253 [2023-09-07 16:54:20,375 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 16:54:20,375 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 16:54:20,376 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 16:54:20,376 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 16:54:20,554 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 16:54:20,554 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 16:54:20,557 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:54:20,557 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 16:54:20,563 ERROR][11][wan-tasty-kitalpha][T_0] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 16:54:20,564 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:54:20,592 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:54:20,619 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:54:20,622 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:54:20,629 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:54:20,629 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694105660.5910215, 'te': 1694105660.597224, 'initial_status': True} ====================================================================================================  [2023-09-07 16:54:20,631 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:54:20.631368 [2023-09-07 16:54:20,635 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:54:20,636 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:54:28,838 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105652, schain_finish_ts: 0 [2023-09-07 16:54:34,977 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:54:34,979 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:54:34,981 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:54:34,982 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:54:34,983 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:54:34,983 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:54:34,989 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:54:34,990 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 16:54:35,013 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:54:35,270 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:54:35,403 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:54:35,505 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:54:35,506 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2483 [2023-09-07 16:54:35,507 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:54:35,507 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:54:35,507 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:54:35,507 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:54:35,508 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:54:35,513 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:54:35,515 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:54:35.515897 [2023-09-07 16:54:35,744 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:54:35,770 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:54:35,831 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:54:35,836 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:54:35,849 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:54:35,850 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694105675.7687423, 'te': 1694105675.7748137, 'initial_status': True} ====================================================================================================  [2023-09-07 16:54:35,855 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:54:35.855644 [2023-09-07 16:54:35,865 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:54:35,868 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:54:45,148 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:54:45,149 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:54:45,152 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:54:45.152884 [2023-09-07 16:54:45,163 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:54:45,163 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:54:45,170 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:54:45,175 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694105685.1615562, 'te': 1694105685.1751933, 'initial_status': True} ====================================================================================================  [2023-09-07 16:54:45,177 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:54:45.177566 [2023-09-07 16:54:45,183 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:54:45,183 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:54:47,300 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105676, schain_finish_ts: 1693742304 [2023-09-07 16:54:49,300 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105676, schain_finish_ts: 1693742004 [2023-09-07 16:54:49,845 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 23 -> 24 [2023-09-07 16:54:52,858 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105688, schain_finish_ts: 1692377364 [2023-09-07 16:54:57,668 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:54:57,668 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:54:57,669 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:54:57,669 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 16:54:57,670 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 16:54:57,670 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:54:57,672 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:54:57,675 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 16:54:57,706 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:54:57,940 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:54:58,213 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:54:58,346 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:54:58,348 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2491 [2023-09-07 16:54:58,349 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:54:58,349 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:54:58,349 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:54:58,350 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:54:58,350 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:54:58,359 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:54:58,361 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:54:58.361271 [2023-09-07 16:54:58,509 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:54:58,545 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:54:58,579 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:54:58,581 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:54:58,589 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:54:58,590 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694105698.5434098, 'te': 1694105698.5504236, 'initial_status': True} ====================================================================================================  [2023-09-07 16:54:58,593 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:54:58.593182 [2023-09-07 16:54:58,601 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:54:58,602 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:55:03,068 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105700, schain_finish_ts: 1693412520 [2023-09-07 16:55:03,575 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:55:03,575 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:55:03,576 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 16:55:04,315 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:55:04,315 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:55:04,318 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:55:04.318283 [2023-09-07 16:55:04,328 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:55:04,328 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:55:04,334 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:55:04,339 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694105704.3270564, 'te': 1694105704.3396134, 'initial_status': True} ====================================================================================================  [2023-09-07 16:55:04,342 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:55:04.342344 [2023-09-07 16:55:04,347 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:55:04,348 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:55:05,671 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:55:05,672 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:55:05,672 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:55:05,672 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:55:05,672 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:55:05,673 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:55:05,674 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:55:05,676 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 16:55:05,703 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:55:05,891 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:55:06,026 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:55:06,128 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:55:06,129 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2468 [2023-09-07 16:55:06,130 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:55:06,130 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:55:06,130 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:55:06,130 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:55:06,131 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:55:06,138 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:55:06,140 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:55:06.140598 [2023-09-07 16:55:06,404 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:55:06,434 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:55:06,481 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:55:06,485 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:55:06,495 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:55:06,495 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694105706.4328942, 'te': 1694105706.4395313, 'initial_status': True} ====================================================================================================  [2023-09-07 16:55:06,497 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:55:06.497418 [2023-09-07 16:55:06,502 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:55:06,503 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:55:06,802 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:55:06,802 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:55:06,806 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:55:06,806 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:55:06,807 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:55:06,807 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:55:06,815 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:55:06,815 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 16:55:06,835 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:55:07,010 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:55:07,013 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:55:07,014 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:55:07,014 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:55:07,014 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:55:07,015 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 16:55:07,040 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105700, schain_finish_ts: 1692637668 [2023-09-07 16:55:07,046 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:55:07,056 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:55:07,206 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:55:07,347 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:55:07,348 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2486 [2023-09-07 16:55:07,348 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:55:07,349 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:55:07,349 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:55:07,349 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:55:07,350 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:55:07,356 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:55:07,360 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:55:07.360917 [2023-09-07 16:55:07,381 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:55:07,568 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:55:07,599 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:55:07,639 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:55:07,642 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:55:07,653 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:55:07,654 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694105707.5965211, 'te': 1694105707.6048043, 'initial_status': True} ====================================================================================================  [2023-09-07 16:55:07,656 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:55:07.656886 [2023-09-07 16:55:07,662 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:55:07,663 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:55:07,713 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:55:07,812 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:55:07,813 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2483 [2023-09-07 16:55:07,814 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:55:07,814 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:55:07,814 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:55:07,815 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:55:07,815 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:55:07,821 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:55:07,823 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:55:07.823410 [2023-09-07 16:55:07,958 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:55:07,992 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:55:08,032 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:55:08,034 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:55:08,044 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:55:08,045 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694105707.9890027, 'te': 1694105707.997584, 'initial_status': True} ====================================================================================================  [2023-09-07 16:55:08,047 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:55:08.047264 [2023-09-07 16:55:08,053 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:55:08,054 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:55:11,654 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:55:11,654 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:55:11,656 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:55:11.656760 [2023-09-07 16:55:11,666 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:55:11,666 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:55:11,673 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:55:11,677 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694105711.6647806, 'te': 1694105711.6776016, 'initial_status': True} ====================================================================================================  [2023-09-07 16:55:11,680 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:55:11.680521 [2023-09-07 16:55:11,685 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:55:11,685 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:55:12,098 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 16:55:12,098 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:55:12,102 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:55:12,102 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:55:12.102474 [2023-09-07 16:55:12,102 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:55:12,103 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:55:12,103 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:55:12,103 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:55:12,106 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:55:12,108 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:55:12,108 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 16:55:12,112 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 16:55:12,139 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:55:12,159 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 16:55:12,160 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 16:55:12,380 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:55:12,669 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:55:12,818 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:55:12,819 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2484 [2023-09-07 16:55:12,819 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:55:12,819 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:55:12,820 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:55:12,820 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:55:12,821 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:55:12,826 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:55:12,828 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:55:12.828730 [2023-09-07 16:55:12,967 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:55:12,995 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:55:13,029 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:55:13,031 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:55:13,039 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:55:13,039 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694105712.9931643, 'te': 1694105712.9994905, 'initial_status': True} ====================================================================================================  [2023-09-07 16:55:13,041 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:55:13.041351 [2023-09-07 16:55:13,046 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:55:13,046 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:55:16,460 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:55:16,460 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:55:16,463 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:55:16.463539 [2023-09-07 16:55:16,472 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:55:16,472 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:55:16,478 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:55:16,483 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694105716.4707644, 'te': 1694105716.4829857, 'initial_status': True} ====================================================================================================  [2023-09-07 16:55:16,485 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:55:16.485265 [2023-09-07 16:55:16,489 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:55:16,491 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:55:22,682 INFO][1][MainProcess][MainThread] - core.schains.process_manager:57 - Process manager started [2023-09-07 16:55:22,793 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 9 -> 64 [2023-09-07 16:55:27,291 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:55:27,291 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:55:27,293 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:55:27.293512 [2023-09-07 16:55:27,302 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:55:27,302 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:55:27,309 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:55:27,314 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694105727.3002276, 'te': 1694105727.3146656, 'initial_status': True} ====================================================================================================  [2023-09-07 16:55:27,316 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:55:27.316618 [2023-09-07 16:55:27,322 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:55:27,322 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:55:27,326 INFO][1][MainProcess][MainThread] - core.schains.notifications:35 - Node account has 406869999689179237 WEI [2023-09-07 16:55:27,327 INFO][1][MainProcess][MainThread] - tools.notifications.messages:188 - Saving new balance state 3007 1 [2023-09-07 16:55:27,328 INFO][1][MainProcess][MainThread] - core.schains.process_manager:95 - Fetching schains to monitor... [2023-09-07 16:55:43,937 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105736, schain_finish_ts: 1692377364 [2023-09-07 16:55:51,194 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105748, schain_finish_ts: 1693412520 [2023-09-07 16:55:53,012 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105748, schain_finish_ts: 1693742304 [2023-09-07 16:56:01,134 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105748, schain_finish_ts: 1692637668 [2023-09-07 16:56:07,143 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:56:07,143 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:56:07,143 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:56:07,144 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:56:07,144 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:56:07,144 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:56:07,144 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:56:07,146 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 16:56:07,172 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:56:07,360 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:56:07,505 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:56:07,606 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:56:07,607 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2469 [2023-09-07 16:56:07,608 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:56:07,608 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:56:07,609 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:56:07,609 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:56:07,609 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:56:07,615 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:56:07,617 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:56:07.617706 [2023-09-07 16:56:07,753 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:56:07,782 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:56:07,822 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:56:07,824 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:56:07,832 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:56:07,832 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694105767.780859, 'te': 1694105767.7898748, 'initial_status': True} ====================================================================================================  [2023-09-07 16:56:07,834 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:56:07.834118 [2023-09-07 16:56:07,839 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:56:07,840 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:56:24,778 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105772, schain_finish_ts: 0 [2023-09-07 16:56:27,032 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:56:27,033 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:56:27,033 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:56:27,033 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:56:27,034 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:56:27,034 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 16:56:27,060 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:56:27,317 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105784, schain_finish_ts: 1693742004 [2023-09-07 16:56:27,322 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:56:27,470 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:56:27,573 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:56:27,574 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2492 [2023-09-07 16:56:27,575 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:56:27,575 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:56:27,575 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:56:27,576 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:56:27,576 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:56:27,581 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:56:27,583 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:56:27.583689 [2023-09-07 16:56:27,726 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:56:27,753 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:56:27,780 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:56:27,782 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:56:27,789 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:56:27,789 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694105787.7514527, 'te': 1694105787.7582228, 'initial_status': True} ====================================================================================================  [2023-09-07 16:56:27,790 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:56:27.790963 [2023-09-07 16:56:27,794 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:56:27,795 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:56:35,849 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:56:35,850 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:56:35,850 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:56:35,850 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:56:35,850 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:56:35,851 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:56:35,854 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:56:35,855 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 16:56:35,878 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:56:36,054 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:56:36,294 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:56:36,390 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:56:36,391 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2487 [2023-09-07 16:56:36,392 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:56:36,392 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:56:36,392 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:56:36,393 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:56:36,393 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:56:36,398 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:56:36,400 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:56:36.400878 [2023-09-07 16:56:36,530 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:56:36,557 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:56:36,587 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:56:36,588 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:56:36,597 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:56:36,597 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694105796.5552058, 'te': 1694105796.5617597, 'initial_status': True} ====================================================================================================  [2023-09-07 16:56:36,599 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:56:36.599135 [2023-09-07 16:56:36,603 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:56:36,603 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:56:43,457 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:56:43,457 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:56:43,459 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:56:43,459 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:56:43,460 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:56:43,460 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:56:43,461 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:56:43,462 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 16:56:43,489 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:56:43,765 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:56:43,951 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:56:44,092 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:56:44,093 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2485 [2023-09-07 16:56:44,094 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:56:44,094 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:56:44,095 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:56:44,095 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:56:44,095 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:56:44,101 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:56:44,103 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:56:44.103966 [2023-09-07 16:56:44,350 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:56:44,380 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:56:44,411 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:56:44,413 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:56:44,421 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:56:44,422 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694105804.3784566, 'te': 1694105804.385338, 'initial_status': True} ====================================================================================================  [2023-09-07 16:56:44,424 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:56:44.424005 [2023-09-07 16:56:44,428 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:56:44,429 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:56:50,667 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105808, schain_finish_ts: 1692377364 [2023-09-07 16:56:53,393 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:56:53,394 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:56:53,396 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:56:53.396049 [2023-09-07 16:56:53,404 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:56:53,404 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:56:53,411 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:56:53,416 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694105813.402223, 'te': 1694105813.4164379, 'initial_status': True} ====================================================================================================  [2023-09-07 16:56:53,419 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:56:53.419013 [2023-09-07 16:56:53,426 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:56:53,426 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:57:01,762 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:57:01,762 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:57:01,763 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:57:01,763 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:57:01,763 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:57:01,763 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 16:57:01,787 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:57:02,090 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:57:02,220 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:57:02,317 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:57:02,317 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2484 [2023-09-07 16:57:02,318 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:57:02,318 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:57:02,319 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:57:02,319 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:57:02,319 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:57:02,325 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:57:02,327 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:57:02.327409 [2023-09-07 16:57:02,469 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:57:02,495 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:57:02,526 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:57:02,528 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:57:02,535 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:57:02,536 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694105822.4934149, 'te': 1694105822.5009215, 'initial_status': True} ====================================================================================================  [2023-09-07 16:57:02,538 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:57:02.538460 [2023-09-07 16:57:02,543 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:57:02,543 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:57:16,224 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:57:16,224 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:57:16,226 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:57:16.226789 [2023-09-07 16:57:16,239 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:57:16,240 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:57:16,249 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:57:16,257 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694105836.2355325, 'te': 1694105836.2571023, 'initial_status': True} ====================================================================================================  [2023-09-07 16:57:16,259 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:57:16.259090 [2023-09-07 16:57:16,264 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:57:16,265 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:57:18,171 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:57:18,172 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:57:18,172 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 16:57:19,825 INFO][1][MainProcess][MainThread] - core.schains.process_manager:109 - Get leaving_history for node ... [2023-09-07 16:57:20,755 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:57:20,755 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:57:20,756 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:57:20,756 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:57:20,756 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:57:20,757 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:57:20,757 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:57:20,759 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 16:57:20,796 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:57:21,007 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:57:21,162 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:57:21,284 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:57:21,285 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2470 [2023-09-07 16:57:21,286 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:57:21,286 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:57:21,286 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:57:21,287 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:57:21,287 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:57:21,294 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:57:21,296 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:57:21.296541 [2023-09-07 16:57:21,555 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:57:21,585 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:57:21,614 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:57:21,616 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:57:21,624 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:57:21,624 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694105841.5832343, 'te': 1694105841.590327, 'initial_status': True} ====================================================================================================  [2023-09-07 16:57:21,626 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:57:21.626383 [2023-09-07 16:57:21,633 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:57:21,634 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:57:23,868 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105832, schain_finish_ts: 1693742304 [2023-09-07 16:57:24,097 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105832, schain_finish_ts: 1693412520 [2023-09-07 16:57:25,821 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 35 -> 16 [2023-09-07 16:57:27,829 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105832, schain_finish_ts: 1692637668 [2023-09-07 16:57:33,093 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:57:33,094 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:57:33,096 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:57:33.096464 [2023-09-07 16:57:33,111 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:57:33,111 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:57:33,121 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:57:33,127 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694105853.1080675, 'te': 1694105853.12758, 'initial_status': True} ====================================================================================================  [2023-09-07 16:57:33,129 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:57:33.129926 [2023-09-07 16:57:33,135 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:57:33,135 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:57:39,995 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105832, schain_finish_ts: 1692377616 [2023-09-07 16:57:41,807 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:57:41,807 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:57:41,807 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:57:41,808 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:57:41,808 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:57:41,808 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:57:41,810 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:57:41,810 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 16:57:41,834 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:57:42,001 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:57:42,162 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:57:42,262 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:57:42,263 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2488 [2023-09-07 16:57:42,263 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:57:42,263 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:57:42,264 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:57:42,264 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:57:42,264 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:57:42,270 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:57:42,271 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:57:42.271903 [2023-09-07 16:57:42,422 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:57:42,451 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:57:42,493 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:57:42,496 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:57:42,505 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:57:42,505 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694105862.4490783, 'te': 1694105862.458212, 'initial_status': True} ====================================================================================================  [2023-09-07 16:57:42,507 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:57:42.507284 [2023-09-07 16:57:42,514 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:57:42,514 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:57:43,056 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:57:43,056 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:57:43,057 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:57:43,057 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:57:43,057 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:57:43,060 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:57:43,061 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:57:43,062 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 16:57:43,093 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:57:43,217 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:57:43,217 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:57:43,217 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:57:43,217 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:57:43,218 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:57:43,218 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 16:57:43,239 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:57:43,301 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:57:43,466 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:57:43,609 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:57:43,803 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:57:43,805 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2486 [2023-09-07 16:57:43,805 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:57:43,805 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:57:43,806 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:57:43,806 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:57:43,806 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:57:43,814 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:57:43,822 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:57:43.822250 [2023-09-07 16:57:43,909 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:57:44,031 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:57:44,032 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2493 [2023-09-07 16:57:44,033 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:57:44,033 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:57:44,034 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:57:44,034 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:57:44,034 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:57:44,043 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:57:44,045 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:57:44.045617 [2023-09-07 16:57:44,086 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:57:44,111 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:57:44,148 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:57:44,152 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:57:44,162 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:57:44,163 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694105864.110062, 'te': 1694105864.1177409, 'initial_status': True} ====================================================================================================  [2023-09-07 16:57:44,166 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:57:44.166932 [2023-09-07 16:57:44,173 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:57:44,174 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:57:44,258 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:57:44,294 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:57:44,337 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:57:44,339 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:57:44,347 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:57:44,347 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694105864.2894676, 'te': 1694105864.3008587, 'initial_status': True} ====================================================================================================  [2023-09-07 16:57:44,349 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:57:44.349170 [2023-09-07 16:57:44,353 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:57:44,353 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:57:49,074 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105832, schain_finish_ts: 1693742004 [2023-09-07 16:57:49,746 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:57:49,746 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:57:49,748 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:57:49.748758 [2023-09-07 16:57:49,757 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:57:49,757 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:57:49,763 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:57:49,769 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694105869.7555816, 'te': 1694105869.7694538, 'initial_status': True} ====================================================================================================  [2023-09-07 16:57:49,771 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:57:49.771519 [2023-09-07 16:57:49,776 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:57:49,776 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:57:52,307 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105868, schain_finish_ts: 1692377364 [2023-09-07 16:57:59,364 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105868, schain_finish_ts: 0 [2023-09-07 16:57:59,467 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105868, schain_finish_ts: 0 [2023-09-07 16:58:02,804 INFO][1][MainProcess][MainThread] - core.schains.process_manager:117 - Got leaving sChains for the node: [] [2023-09-07 16:58:02,805 INFO][1][MainProcess][MainThread] - core.schains.process_manager:101 -  ==================================================================================================== Monitoring sChains Node ID: 16 sChains on node: [{'name': 'portly-passionate-sirius', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666281384, 'startBlock': 7803380, 'deposit': 0, 'index': 1, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x0000017723ca2', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'curly-red-alterf', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1676058936, 'startBlock': 8469646, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000001fe6cdfa', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'howling-hoedus', 'mainnetOwner': '0x1026467C91fdA40aCc7e6935d1E52f67274FD669', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666181400, 'startBlock': 7796660, 'deposit': 0, 'index': 0, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0xe5a600f4c2608', 'multitransactionMode': False, 'thresholdEncryption': False, 'active': True}, {'name': 'green-giddy-denebola', 'mainnetOwner': '0x8e835d5d9E28ffD259E8Fa8988Cacfc333A582B4', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674565164, 'startBlock': 8368049, 'deposit': 0, 'index': 4, 'generation': 1, 'originator': '0xd2001DAb6898127Be2F167B548691C87251D13C3', 'chainId': '0x00000585eb4b1', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'wan-tasty-kitalpha', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674568392, 'startBlock': 8368271, 'deposit': 0, 'index': 5, 'generation': 1, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x00000152e7be9', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'submissive-well-off-rotanev', 'mainnetOwner': '0x487360568fE5Ee4AE5c75782D5F63e9935B0E512', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1692361104, 'startBlock': 9539053, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000003791fa10', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}] Number of sChains on node: 6 Empty sChain structs: 1 ====================================================================================================  [2023-09-07 16:58:08,201 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:58:08,202 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:58:08,204 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:58:08.204404 [2023-09-07 16:58:08,212 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:58:08,212 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:58:08,218 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:58:08,225 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694105888.2106402, 'te': 1694105888.2248878, 'initial_status': True} ====================================================================================================  [2023-09-07 16:58:08,227 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:58:08.227209 [2023-09-07 16:58:08,232 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:58:08,233 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:58:13,911 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:58:13,911 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain portly-passionate-sirius - Process is running: PID = 7 [2023-09-07 16:58:15,218 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:58:15,218 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:58:15,219 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:58:15,219 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:58:15,219 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:58:15,219 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:58:15,221 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:58:15,222 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 16:58:15,253 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:58:15,550 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:58:15,791 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:58:15,973 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:58:15,974 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2471 [2023-09-07 16:58:15,975 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:58:15,975 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:58:15,976 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:58:15,976 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:58:15,976 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:58:15,987 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:58:15,990 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:58:15.990203 [2023-09-07 16:58:16,249 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:58:16,285 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:58:16,343 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:58:16,347 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:58:16,361 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:58:16,362 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694105896.2831552, 'te': 1694105896.2922966, 'initial_status': True} ====================================================================================================  [2023-09-07 16:58:16,365 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:58:16.365652 [2023-09-07 16:58:16,371 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:58:16,372 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:58:25,412 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105892, schain_finish_ts: 1693742304 [2023-09-07 16:58:25,546 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:58:25,547 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain curly-red-alterf - Process is running: PID = 8 [2023-09-07 16:58:26,738 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105904, schain_finish_ts: 1693412520 [2023-09-07 16:58:26,840 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:58:26,840 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:58:26,840 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 16:58:28,548 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:58:28,548 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:58:28,548 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:58:28,548 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:58:28,549 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:58:28,549 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 16:58:28,584 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:58:28,834 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:58:29,017 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:58:29,121 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:58:29,122 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2485 [2023-09-07 16:58:29,122 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:58:29,123 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:58:29,123 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:58:29,123 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:58:29,123 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:58:29,130 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:58:29,131 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:58:29.131933 [2023-09-07 16:58:29,304 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:58:29,330 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:58:29,365 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:58:29,369 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:58:29,379 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:58:29,380 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694105909.3283067, 'te': 1694105909.3360853, 'initial_status': True} ====================================================================================================  [2023-09-07 16:58:29,383 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:58:29.383653 [2023-09-07 16:58:29,389 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:58:29,390 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:58:29,641 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:58:29,641 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain howling-hoedus - Process is running: PID = 9 [2023-09-07 16:58:31,726 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105904, schain_finish_ts: 1692637668 [2023-09-07 16:58:35,212 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:58:35,212 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:58:35,215 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:58:35.215381 [2023-09-07 16:58:35,224 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:58:35,225 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:58:35,233 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:58:35,244 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694105915.2229013, 'te': 1694105915.2441447, 'initial_status': True} ====================================================================================================  [2023-09-07 16:58:35,246 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:58:35.246690 [2023-09-07 16:58:35,251 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:58:35,252 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:58:36,002 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:58:36,002 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:58:36,002 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:58:36,003 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 16:58:36,003 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 16:58:36,003 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 16:58:36,005 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 16:58:36,008 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 16:58:36,028 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 16:58:36,195 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:58:36,195 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain green-giddy-denebola - Process is running: PID = 10 [2023-09-07 16:58:36,228 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:58:36,532 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:58:36,683 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:58:36,685 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2487 [2023-09-07 16:58:36,686 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 16:58:36,686 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:58:36,686 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:58:36,687 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:58:36,687 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:58:36,693 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:58:36,695 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:58:36.695631 [2023-09-07 16:58:36,842 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:58:36,878 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 16:58:36,913 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 16:58:36,916 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 16:58:36,925 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:58:36,925 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694105916.875982, 'te': 1694105916.8841352, 'initial_status': True} ====================================================================================================  [2023-09-07 16:58:36,927 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:58:36.927295 [2023-09-07 16:58:36,934 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:58:36,935 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:58:41,599 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:58:41,599 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain wan-tasty-kitalpha - Process is running: PID = 11 [2023-09-07 16:58:48,518 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:58:48,519 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:58:48,519 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:58:48,519 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 16:58:48,519 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 16:58:48,520 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 16:58:48,526 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 16:58:48,526 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 16:58:48,552 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 16:58:48,859 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:58:48,988 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:58:49,102 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:58:49,104 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2489 [2023-09-07 16:58:49,105 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 16:58:49,105 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:58:49,105 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:58:49,106 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:58:49,106 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:58:49,116 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:58:49,118 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:58:49.118713 [2023-09-07 16:58:49,261 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:58:49,299 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 16:58:49,335 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 16:58:49,337 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 16:58:49,347 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:58:49,347 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694105929.2973576, 'te': 1694105929.3050473, 'initial_status': True} ====================================================================================================  [2023-09-07 16:58:49,349 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:58:49.349302 [2023-09-07 16:58:49,355 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:58:49,355 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:58:53,233 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 16:58:53,233 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain submissive-well-off-rotanev - Process is running: PID = 24912 [2023-09-07 16:58:53,233 INFO][1][MainProcess][MainThread] - core.schains.process_manager:88 - Process manager procedure finished [2023-09-07 16:58:53,233 INFO][1][MainProcess][MainThread] - __main__:65 - Sleeping for 90s after run_process_manager [2023-09-07 16:58:57,275 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:58:57,276 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:58:57,276 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:58:57,276 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 16:58:57,276 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 16:58:57,276 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 16:58:57,295 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 16:58:57,467 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:58:57,596 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:58:57,698 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:58:57,699 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2494 [2023-09-07 16:58:57,700 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 16:58:57,700 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:58:57,700 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:58:57,700 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:58:57,701 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:58:57,707 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:58:57,709 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:58:57.709320 [2023-09-07 16:58:57,854 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:58:57,879 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 16:58:57,905 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 16:58:57,907 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 16:58:57,914 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:58:57,914 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694105937.8773313, 'te': 1694105937.8837652, 'initial_status': True} ====================================================================================================  [2023-09-07 16:58:57,916 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 16:58:57.916255 [2023-09-07 16:58:57,920 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:58:57,920 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:59:04,654 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:59:04,654 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:59:04,657 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:59:04.657081 [2023-09-07 16:59:04,664 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 16:59:04,664 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 16:59:04,669 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 16:59:04,674 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694105944.6630087, 'te': 1694105944.6739721, 'initial_status': True} ====================================================================================================  [2023-09-07 16:59:04,675 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:59:04.675722 [2023-09-07 16:59:04,680 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:59:04,680 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:59:08,300 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105940, schain_finish_ts: 1693742004 [2023-09-07 16:59:10,074 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105940, schain_finish_ts: 1692377364 [2023-09-07 16:59:12,922 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105940, schain_finish_ts: 0 [2023-09-07 16:59:15,477 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 55 -> 63 [2023-09-07 16:59:19,212 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105952, schain_finish_ts: 1692637668 [2023-09-07 16:59:28,668 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:59:28,668 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:59:28,670 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:59:28.670210 [2023-09-07 16:59:28,679 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 16:59:28,679 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 16:59:28,686 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 16:59:28,691 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694105968.6773937, 'te': 1694105968.691434, 'initial_status': True} ====================================================================================================  [2023-09-07 16:59:28,693 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 16:59:28.693472 [2023-09-07 16:59:28,698 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:59:28,698 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:59:29,312 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:59:29,312 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:59:29,313 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:59:29,313 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 16:59:29,313 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 16:59:29,313 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 16:59:29,315 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 16:59:29,316 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 16:59:29,347 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 16:59:29,569 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:59:29,754 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:59:29,881 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:59:29,883 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2472 [2023-09-07 16:59:29,884 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 16:59:29,884 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:59:29,885 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:59:29,886 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:59:29,886 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:59:29,893 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:59:29,900 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:59:29.900699 [2023-09-07 16:59:30,315 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:59:30,342 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 16:59:30,376 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 16:59:30,378 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 16:59:30,386 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:59:30,387 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694105970.3403986, 'te': 1694105970.348327, 'initial_status': True} ====================================================================================================  [2023-09-07 16:59:30,389 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 16:59:30.389464 [2023-09-07 16:59:30,394 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:59:30,395 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 16:59:39,676 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 16:59:39,676 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 16:59:39,678 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:59:39.678870 [2023-09-07 16:59:39,687 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 16:59:39,688 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 16:59:39,695 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 16:59:39,700 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694105979.6859112, 'te': 1694105979.7006836, 'initial_status': True} ====================================================================================================  [2023-09-07 16:59:39,702 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 16:59:39.702952 [2023-09-07 16:59:39,708 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 16:59:39,708 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 16:59:39,796 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105964, schain_finish_ts: 1693742304 [2023-09-07 16:59:40,180 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694105964, schain_finish_ts: 1693412520 [2023-09-07 16:59:41,317 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 16:59:41,317 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:59:41,317 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 16:59:45,977 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 16:59:45,977 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 16:59:45,977 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 16:59:45,978 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 16:59:45,978 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 16:59:45,978 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 16:59:46,007 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 16:59:46,412 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:59:46,546 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:59:46,644 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:59:46,645 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2486 [2023-09-07 16:59:46,646 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 16:59:46,646 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 16:59:46,646 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 16:59:46,647 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:59:46,647 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 16:59:46,654 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 16:59:46,656 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:59:46.656055 [2023-09-07 16:59:46,794 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 16:59:46,821 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 16:59:46,852 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 16:59:46,853 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 16:59:46,863 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 16:59:46,863 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694105986.8194408, 'te': 1694105986.8262422, 'initial_status': True} ====================================================================================================  [2023-09-07 16:59:46,865 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 16:59:46.865091 [2023-09-07 16:59:46,869 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 16:59:46,870 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:00:00,023 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:00:00,024 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:00:00,024 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:00:00,024 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:00:00,024 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:00:00,025 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:00:00,029 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:00:00,030 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:00:00,052 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:00:00,252 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:00:00,388 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:00:00,487 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:00:00,488 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2488 [2023-09-07 17:00:00,489 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:00:00,489 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:00:00,489 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:00:00,489 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:00:00,490 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:00:00,495 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:00:00,497 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:00:00.497441 [2023-09-07 17:00:00,657 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:00:00,684 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:00:00,720 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:00:00,723 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:00:00,738 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:00:00,739 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694106000.6825583, 'te': 1694106000.6915696, 'initial_status': True} ====================================================================================================  [2023-09-07 17:00:00,743 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:00:00.743495 [2023-09-07 17:00:00,750 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:00:00,750 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:00:02,831 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:00:02,831 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:00:02,832 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:00:02,832 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:00:02,832 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:00:02,832 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 17:00:02,865 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:00:02,895 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:00:02,895 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:00:02,895 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:00:02,895 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:00:02,896 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:00:02,896 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:00:02,896 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:00:02,897 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:00:02,936 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:00:03,157 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:00:03,289 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:00:03,573 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:00:03,694 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:00:03,696 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2495 [2023-09-07 17:00:03,696 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:00:03,697 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:00:03,697 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:00:03,697 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:00:03,698 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:00:03,707 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:00:03,709 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:00:03.709864 [2023-09-07 17:00:03,755 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:00:03,893 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:00:03,895 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2490 [2023-09-07 17:00:03,896 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:00:03,896 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:00:03,897 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:00:03,897 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:00:03,897 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:00:03,908 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:00:03,911 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:00:03.911744 [2023-09-07 17:00:03,916 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:00:03,947 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:00:03,983 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:00:03,986 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:00:03,995 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:00:03,996 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694106003.944606, 'te': 1694106003.9527264, 'initial_status': True} ====================================================================================================  [2023-09-07 17:00:03,997 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:00:03.997968 [2023-09-07 17:00:04,004 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:00:04,005 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:00:04,171 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:00:04,211 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:00:04,239 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:00:04,241 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:00:04,249 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:00:04,250 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694106004.209546, 'te': 1694106004.2158377, 'initial_status': True} ====================================================================================================  [2023-09-07 17:00:04,251 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:00:04.251704 [2023-09-07 17:00:04,257 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:00:04,259 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:00:12,706 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106000, schain_finish_ts: 1692377364 [2023-09-07 17:00:21,075 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:00:21,076 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:00:21,079 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:00:21.079875 [2023-09-07 17:00:21,090 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:00:21,091 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:00:21,098 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:00:21,103 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694106021.0877335, 'te': 1694106021.103423, 'initial_status': True} ====================================================================================================  [2023-09-07 17:00:21,106 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:00:21.106423 [2023-09-07 17:00:21,111 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:00:21,112 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:00:23,303 INFO][1][MainProcess][MainThread] - core.schains.cleaner:65 - Cleaner process started [2023-09-07 17:00:23,340 INFO][30555][cleaner][MainThread] - core.schains.cleaner:110 - Cleaner procedure started. [2023-09-07 17:00:35,028 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:00:35,029 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:00:35,029 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:00:35,029 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:00:35,029 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:00:35,030 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:00:35,032 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:00:35,032 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:00:35,086 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:00:35,290 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:00:35,455 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:00:35,620 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:00:35,622 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2473 [2023-09-07 17:00:35,624 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:00:35,624 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:00:35,624 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:00:35,625 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:00:35,626 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:00:35,640 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:00:35,642 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:00:35.642752 [2023-09-07 17:00:35,953 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:00:35,985 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:00:36,031 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:00:36,033 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:00:36,041 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:00:36,041 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694106035.983061, 'te': 1694106035.9920785, 'initial_status': True} ====================================================================================================  [2023-09-07 17:00:36,043 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:00:36.043180 [2023-09-07 17:00:36,048 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:00:36,048 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:00:39,173 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:00:39,174 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:00:39,176 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:00:39.176633 [2023-09-07 17:00:39,185 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:00:39,186 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:00:39,193 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:00:39,198 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694106039.183595, 'te': 1694106039.1985865, 'initial_status': True} ====================================================================================================  [2023-09-07 17:00:39,201 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:00:39.201255 [2023-09-07 17:00:39,208 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:00:39,208 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:00:44,909 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106036, schain_finish_ts: 1693742304 [2023-09-07 17:00:46,644 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106036, schain_finish_ts: 1693742004 [2023-09-07 17:00:49,743 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106048, schain_finish_ts: 0 [2023-09-07 17:00:52,693 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106048, schain_finish_ts: 1692637668 [2023-09-07 17:00:55,945 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:00:55,945 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:00:55,945 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:00:55,946 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:00:55,946 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:00:55,946 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 17:00:55,972 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:00:55,995 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 14 -> 11 [2023-09-07 17:00:56,174 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:00:56,312 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:00:56,424 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:00:56,425 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2496 [2023-09-07 17:00:56,426 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:00:56,426 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:00:56,426 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:00:56,426 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:00:56,427 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:00:56,433 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:00:56,435 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:00:56.435245 [2023-09-07 17:00:56,574 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:00:56,634 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:00:56,689 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:00:56,699 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:00:56,711 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:00:56,711 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694106056.6328018, 'te': 1694106056.640122, 'initial_status': True} ====================================================================================================  [2023-09-07 17:00:56,713 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:00:56.713786 [2023-09-07 17:00:56,729 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:00:56,730 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:00:56,891 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:00:56,892 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:00:56,894 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:00:56.894556 [2023-09-07 17:00:56,906 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:00:56,906 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:00:56,914 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:00:56,923 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694106056.9030092, 'te': 1694106056.9234178, 'initial_status': True} ====================================================================================================  [2023-09-07 17:00:56,929 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:00:56.929011 [2023-09-07 17:00:56,935 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:00:56,935 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:00:58,590 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:00:58,591 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:00:58,592 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:01:02,384 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106048, schain_finish_ts: 1693412520 [2023-09-07 17:01:02,434 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:01:02,435 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:01:02,437 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:01:02.437409 [2023-09-07 17:01:02,454 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:01:02,454 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:01:02,468 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:01:02,473 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694106062.450838, 'te': 1694106062.472912, 'initial_status': True} ====================================================================================================  [2023-09-07 17:01:02,475 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:01:02.474989 [2023-09-07 17:01:02,480 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:01:02,480 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:01:06,510 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:01:06,510 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:01:06,511 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:01:06,511 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:01:06,511 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:01:06,511 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:01:06,547 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:01:06,794 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:01:07,001 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:01:07,111 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:01:07,112 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2487 [2023-09-07 17:01:07,113 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:01:07,113 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:01:07,114 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:01:07,114 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:01:07,115 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:01:07,123 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:01:07,126 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:01:07.126319 [2023-09-07 17:01:07,343 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:01:07,375 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:01:07,408 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:01:07,410 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:01:07,418 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:01:07,418 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694106067.3734827, 'te': 1694106067.3840015, 'initial_status': True} ====================================================================================================  [2023-09-07 17:01:07,420 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:01:07.420282 [2023-09-07 17:01:07,424 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:01:07,425 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:01:23,711 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:01:23,711 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:01:23,711 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:01:23,713 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:01:23,714 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:01:23,715 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:01:23,726 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:01:23,726 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 17:01:23,755 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:01:23,977 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:01:24,154 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:01:24,275 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:01:24,276 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2491 [2023-09-07 17:01:24,277 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:01:24,278 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:01:24,279 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:01:24,279 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:01:24,280 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:01:24,287 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:01:24,290 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:01:24.290263 [2023-09-07 17:01:24,469 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:01:24,500 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:01:24,531 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:01:24,533 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:01:24,543 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:01:24,544 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694106084.4980192, 'te': 1694106084.505199, 'initial_status': True} ====================================================================================================  [2023-09-07 17:01:24,545 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:01:24.545897 [2023-09-07 17:01:24,550 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:01:24,551 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:01:26,744 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:01:26,745 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:01:26,745 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:01:26,746 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:01:26,746 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:01:26,753 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:01:26,756 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:01:26,759 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:01:26,799 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:01:27,069 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:01:27,428 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:01:27,565 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:01:27,567 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2489 [2023-09-07 17:01:27,568 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:01:27,569 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:01:27,569 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:01:27,570 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:01:27,571 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:01:27,587 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:01:27,591 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:01:27.591230 [2023-09-07 17:01:27,797 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:01:27,827 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:01:27,866 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:01:27,869 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:01:27,878 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:01:27,879 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694106087.8255315, 'te': 1694106087.8329613, 'initial_status': True} ====================================================================================================  [2023-09-07 17:01:27,881 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:01:27.881489 [2023-09-07 17:01:27,887 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:01:27,888 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:01:38,848 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 17:01:38,848 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 17:01:41,781 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106096, schain_finish_ts: 1692377364 [2023-09-07 17:01:47,884 INFO][30555][cleaner][MainThread] - core.schains.cleaner:116 - sChains on contracts: ['portly-passionate-sirius', 'curly-red-alterf', 'howling-hoedus', 'green-giddy-denebola', 'wan-tasty-kitalpha', '', 'submissive-well-off-rotanev'] sChains on node: ['curly-red-alterf', 'green-giddy-denebola', 'howling-hoedus', 'portly-passionate-sirius', 'submissive-well-off-rotanev', 'wan-tasty-kitalpha'] [2023-09-07 17:01:47,885 INFO][30555][cleaner][MainThread] - core.schains.cleaner:132 - Cleanup procedure finished [2023-09-07 17:01:47,890 INFO][1][MainProcess][MainThread] - core.schains.cleaner:67 - Cleaner process is joined. [2023-09-07 17:01:47,890 INFO][1][MainProcess][MainThread] - core.schains.cleaner:68 - Terminating the process [2023-09-07 17:01:47,890 INFO][1][MainProcess][MainThread] - __main__:70 - Sleeping for 90s after run_cleaner [2023-09-07 17:01:52,389 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106108, schain_finish_ts: 0 [2023-09-07 17:02:00,592 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106108, schain_finish_ts: 1693742004 [2023-09-07 17:02:04,486 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:02:04,486 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:02:04,488 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:02:04.488702 [2023-09-07 17:02:04,496 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:02:04,496 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:02:04,502 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:02:04,506 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694106124.494545, 'te': 1694106124.5065818, 'initial_status': True} ====================================================================================================  [2023-09-07 17:02:04,508 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:02:04.508445 [2023-09-07 17:02:04,512 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:02:04,513 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:02:04,966 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106120, schain_finish_ts: 1693412520 [2023-09-07 17:02:06,883 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106120, schain_finish_ts: 1693742304 [2023-09-07 17:02:12,414 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:02:12,415 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:02:12,417 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:02:12.417455 [2023-09-07 17:02:12,426 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:02:12,427 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:02:12,434 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:02:12,439 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694106132.4243505, 'te': 1694106132.4394553, 'initial_status': True} ====================================================================================================  [2023-09-07 17:02:12,442 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:02:12.442169 [2023-09-07 17:02:12,447 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:02:12,447 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:02:20,349 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:02:20,349 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:02:20,349 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:02:20,350 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:02:20,350 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:02:20,350 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:02:20,353 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:02:20,356 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 17:02:20,395 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:02:20,502 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:02:20,503 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:02:20,503 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 17:02:20,637 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:02:20,889 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:02:20,996 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:02:20,998 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2474 [2023-09-07 17:02:20,998 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:02:20,999 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:02:20,999 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:02:20,999 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:02:21,000 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:02:21,006 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:02:21,008 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:02:21.008118 [2023-09-07 17:02:21,145 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:02:21,194 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:02:21,226 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:02:21,228 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:02:21,235 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:02:21,235 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694106141.1925123, 'te': 1694106141.2022424, 'initial_status': True} ====================================================================================================  [2023-09-07 17:02:21,237 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:02:21.237231 [2023-09-07 17:02:21,242 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:02:21,243 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:02:21,430 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:02:21,431 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:02:21,431 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:02:21,431 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:02:21,431 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:02:21,431 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:02:21,437 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:02:21,437 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:02:21,463 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:02:21,709 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:02:21,891 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:02:21,992 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:02:21,993 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2492 [2023-09-07 17:02:21,994 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:02:21,994 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:02:21,995 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:02:21,995 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:02:21,995 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:02:22,001 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:02:22,003 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:02:22.003449 [2023-09-07 17:02:22,290 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:02:22,327 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:02:22,364 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:02:22,366 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:02:22,375 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:02:22,376 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694106142.325495, 'te': 1694106142.3318276, 'initial_status': True} ====================================================================================================  [2023-09-07 17:02:22,377 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:02:22.377927 [2023-09-07 17:02:22,382 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:02:22,383 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:02:26,426 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:02:26,426 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:02:26,426 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:02:26,427 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:02:26,427 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:02:26,427 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:02:26,453 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:02:26,910 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:02:27,198 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:02:27,349 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:02:27,350 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2488 [2023-09-07 17:02:27,351 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:02:27,351 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:02:27,351 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:02:27,352 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:02:27,352 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:02:27,359 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:02:27,362 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:02:27.362110 [2023-09-07 17:02:27,544 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:02:27,575 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:02:27,606 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:02:27,609 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:02:27,619 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:02:27,619 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694106147.5732908, 'te': 1694106147.5809326, 'initial_status': True} ====================================================================================================  [2023-09-07 17:02:27,621 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:02:27.621691 [2023-09-07 17:02:27,626 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:02:27,627 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:02:30,339 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:02:30,340 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:02:30,340 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:02:30,341 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:02:30,341 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:02:30,341 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 17:02:30,375 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:02:30,551 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:02:30,782 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:02:30,904 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:02:30,905 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2497 [2023-09-07 17:02:30,905 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:02:30,906 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:02:30,906 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:02:30,906 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:02:30,906 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:02:30,913 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:02:30,915 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:02:30.915851 [2023-09-07 17:02:31,082 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:02:31,099 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106132, schain_finish_ts: 1692637668 [2023-09-07 17:02:31,114 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:02:31,154 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:02:31,157 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:02:31,169 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:02:31,170 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694106151.112636, 'te': 1694106151.1191866, 'initial_status': True} ====================================================================================================  [2023-09-07 17:02:31,172 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:02:31.172910 [2023-09-07 17:02:31,179 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:02:31,180 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:02:54,720 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:02:54,720 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:02:54,722 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:02:54.722333 [2023-09-07 17:02:54,729 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:02:54,729 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:02:54,735 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:02:54,740 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694106174.7278981, 'te': 1694106174.7403786, 'initial_status': True} ====================================================================================================  [2023-09-07 17:02:54,742 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:02:54.742138 [2023-09-07 17:02:54,747 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:02:54,747 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:02:54,831 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:02:54,831 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:02:54,833 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:02:54.833075 [2023-09-07 17:02:54,840 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:02:54,840 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:02:54,847 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:02:54,852 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694106174.8387582, 'te': 1694106174.8525116, 'initial_status': True} ====================================================================================================  [2023-09-07 17:02:54,854 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:02:54.854453 [2023-09-07 17:02:54,860 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:02:54,860 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:02:55,386 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:02:55,386 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:02:55,386 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:02:55,386 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:02:55,386 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:02:55,387 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:02:55,390 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:02:55,392 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 17:02:55,410 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:02:55,611 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:02:55,735 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:02:55,829 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:02:55,830 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2490 [2023-09-07 17:02:55,831 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:02:55,831 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:02:55,833 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:02:55,835 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:02:55,836 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:02:55,845 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:02:55,852 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:02:55.852076 [2023-09-07 17:02:56,013 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:02:56,040 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:02:56,067 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:02:56,069 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:02:56,077 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:02:56,078 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694106176.0387778, 'te': 1694106176.0460439, 'initial_status': True} ====================================================================================================  [2023-09-07 17:02:56,079 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:02:56.079916 [2023-09-07 17:02:56,084 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:02:56,085 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:03:04,592 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106168, schain_finish_ts: 0 [2023-09-07 17:03:10,791 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106168, schain_finish_ts: 1692377364 [2023-09-07 17:03:12,035 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106168, schain_finish_ts: 1693412520 [2023-09-07 17:03:13,098 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:03:13,098 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:03:13,100 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:03:13.100930 [2023-09-07 17:03:13,110 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:03:13,110 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:03:13,118 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:03:13,124 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694106193.1088204, 'te': 1694106193.1245031, 'initial_status': True} ====================================================================================================  [2023-09-07 17:03:13,131 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:03:13.130703 [2023-09-07 17:03:13,142 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:03:13,142 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:03:13,679 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106168, schain_finish_ts: 1693742304 [2023-09-07 17:03:14,934 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 17:03:15,109 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:03:15,110 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:03:15,112 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:03:15,112 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:03:15,277 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:03:15,278 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:03:15,282 INFO][8][curly-red-alterf][T_1] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 17:03:15,290 ERROR][8][curly-red-alterf][T_1] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:03:15,291 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:03:17,976 INFO][1][MainProcess][MainThread] - core.schains.process_manager:57 - Process manager started [2023-09-07 17:03:18,201 INFO][1][MainProcess][MainThread] - core.schains.notifications:35 - Node account has 406869999689179237 WEI [2023-09-07 17:03:18,202 INFO][1][MainProcess][MainThread] - tools.notifications.messages:188 - Saving new balance state 3008 1 [2023-09-07 17:03:18,203 INFO][1][MainProcess][MainThread] - core.schains.process_manager:95 - Fetching schains to monitor... [2023-09-07 17:03:23,289 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106192, schain_finish_ts: 1693742004 [2023-09-07 17:03:26,733 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:03:26,733 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:03:26,733 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:03:30,978 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:03:30,978 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:03:30,979 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:03:30,979 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:03:30,979 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:03:30,979 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:03:30,981 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:03:30,981 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 17:03:31,017 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:03:31,220 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:03:31,350 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:03:31,470 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:03:31,472 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2475 [2023-09-07 17:03:31,473 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:03:31,473 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:03:31,474 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:03:31,475 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:03:31,476 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:03:31,487 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:03:31,489 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:03:31.489293 [2023-09-07 17:03:31,663 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:03:31,692 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:03:31,770 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:03:31,789 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:03:31,822 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:03:31,823 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694106211.6909041, 'te': 1694106211.6998348, 'initial_status': True} ====================================================================================================  [2023-09-07 17:03:31,831 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:03:31.831355 [2023-09-07 17:03:31,841 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:03:31,849 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:03:40,585 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 56 -> 60 [2023-09-07 17:03:44,875 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:03:44,875 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:03:44,875 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:03:44,875 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:03:44,876 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:03:44,876 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:03:44,877 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:03:44,877 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 17:03:44,908 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:03:45,093 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:03:45,218 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:03:45,326 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:03:45,327 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2493 [2023-09-07 17:03:45,328 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:03:45,328 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:03:45,328 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:03:45,329 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:03:45,329 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:03:45,335 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:03:45,337 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:03:45.337425 [2023-09-07 17:03:45,468 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:03:45,498 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:03:45,528 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:03:45,530 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:03:45,537 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:03:45,537 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694106225.4949255, 'te': 1694106225.5035512, 'initial_status': True} ====================================================================================================  [2023-09-07 17:03:45,538 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:03:45.538897 [2023-09-07 17:03:45,543 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:03:45,543 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:03:49,560 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:03:49,560 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:03:49,561 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:03:49,561 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:03:49,561 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:03:49,561 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:03:49,583 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:03:49,766 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:03:49,900 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:03:50,036 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:03:50,037 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2498 [2023-09-07 17:03:50,038 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:03:50,038 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:03:50,039 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:03:50,039 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:03:50,039 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:03:50,045 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:03:50,048 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:03:50.047986 [2023-09-07 17:03:50,233 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:03:50,265 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:03:50,293 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:03:50,295 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:03:50,302 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:03:50,303 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694106230.263544, 'te': 1694106230.271373, 'initial_status': True} ====================================================================================================  [2023-09-07 17:03:50,304 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:03:50.304424 [2023-09-07 17:03:50,309 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:03:50,310 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:03:50,655 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:03:50,655 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:03:50,655 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:03:50,655 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 17:03:50,656 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 17:03:50,656 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:03:50,658 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:03:50,658 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:03:50,683 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:03:50,888 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:03:51,022 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:03:51,138 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:03:51,139 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2489 [2023-09-07 17:03:51,140 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:03:51,140 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:03:51,140 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:03:51,140 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:03:51,141 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:03:51,149 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:03:51,151 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:03:51.151778 [2023-09-07 17:03:51,433 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:03:51,460 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:03:51,493 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:03:51,495 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:03:51,505 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:03:51,505 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694106231.4586008, 'te': 1694106231.4655855, 'initial_status': True} ====================================================================================================  [2023-09-07 17:03:51,508 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:03:51.508022 [2023-09-07 17:03:51,513 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:03:51,514 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:03:53,250 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106228, schain_finish_ts: 1692637668 [2023-09-07 17:04:24,782 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106252, schain_finish_ts: 0 [2023-09-07 17:04:26,755 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:04:26,755 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:04:26,755 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:04:26,755 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:04:26,755 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:04:26,755 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:04:26,757 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:04:26,758 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 17:04:26,783 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:04:27,049 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:04:27,177 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:04:27,273 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:04:27,274 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2491 [2023-09-07 17:04:27,275 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:04:27,275 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:04:27,275 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:04:27,275 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:04:27,276 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:04:27,281 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:04:27,283 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:04:27.283322 [2023-09-07 17:04:27,420 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:04:27,447 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:04:27,476 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:04:27,478 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:04:27,485 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:04:27,486 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694106267.4458895, 'te': 1694106267.452473, 'initial_status': True} ====================================================================================================  [2023-09-07 17:04:27,488 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:04:27.488321 [2023-09-07 17:04:27,493 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:04:27,493 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:04:38,152 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:04:38,152 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:04:38,154 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:04:38.154698 [2023-09-07 17:04:38,159 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:04:39,347 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106264, schain_finish_ts: 1693412520 [2023-09-07 17:04:45,093 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106276, schain_finish_ts: 1692377364 [2023-09-07 17:04:45,098 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:04:45,099 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:04:45,100 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:04:45.100831 [2023-09-07 17:04:45,107 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:04:45,108 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:04:45,113 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:04:45,118 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694106285.106498, 'te': 1694106285.1178613, 'initial_status': True} ====================================================================================================  [2023-09-07 17:04:45,119 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:04:45.119719 [2023-09-07 17:04:45,124 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:04:45,124 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:04:45,697 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:04:45,698 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:04:45,701 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:04:45.701287 [2023-09-07 17:04:45,709 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:04:45,709 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:04:45,716 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:04:45,721 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694106285.7080648, 'te': 1694106285.7210972, 'initial_status': True} ====================================================================================================  [2023-09-07 17:04:45,723 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:04:45.723421 [2023-09-07 17:04:45,728 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:04:45,728 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:04:56,385 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106288, schain_finish_ts: 1693742004 [2023-09-07 17:05:03,607 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:05:03,607 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:05:03,607 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:05:03,608 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:05:03,608 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:05:03,608 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:05:03,613 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:05:03,614 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 17:05:03,642 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:05:03,952 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:05:04,203 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:05:04,309 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:05:04,310 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2494 [2023-09-07 17:05:04,311 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:05:04,311 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:05:04,311 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:05:04,311 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:05:04,312 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:05:04,317 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:05:04,320 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:05:04.319992 [2023-09-07 17:05:04,514 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:05:04,544 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:05:04,573 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:05:04,576 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:05:04,582 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:05:04,583 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694106304.542326, 'te': 1694106304.548897, 'initial_status': True} ====================================================================================================  [2023-09-07 17:05:04,584 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:05:04.584896 [2023-09-07 17:05:04,589 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:05:04,589 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:05:07,508 INFO][1][MainProcess][MainThread] - core.schains.process_manager:109 - Get leaving_history for node ... [2023-09-07 17:05:09,558 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:05:09,558 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:05:09,559 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:05:11,453 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:05:11,453 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:05:11,454 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:05:11,454 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:05:11,454 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:05:11,455 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 17:05:11,480 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:05:11,819 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:05:12,009 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:05:12,123 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:05:12,124 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2490 [2023-09-07 17:05:12,124 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:05:12,125 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:05:12,125 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:05:12,125 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:05:12,126 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:05:12,133 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:05:12,136 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:05:12.136195 [2023-09-07 17:05:12,283 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:05:12,309 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:05:12,337 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:05:12,339 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:05:12,346 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:05:12,347 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694106312.3073397, 'te': 1694106312.314194, 'initial_status': True} ====================================================================================================  [2023-09-07 17:05:12,348 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:05:12.348728 [2023-09-07 17:05:12,353 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:05:12,353 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:05:13,122 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:05:13,122 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:05:13,123 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:05:13,123 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:05:13,123 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:05:13,125 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:05:13,131 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:05:13,132 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 17:05:13,156 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:05:13,240 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:05:13,240 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:05:13,242 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:05:13.242641 [2023-09-07 17:05:13,250 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:05:13,250 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:05:13,266 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:05:13,271 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694106313.249093, 'te': 1694106313.2716014, 'initial_status': True} ====================================================================================================  [2023-09-07 17:05:13,273 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:05:13.273563 [2023-09-07 17:05:13,279 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:05:13,279 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:05:13,338 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:05:13,582 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:05:13,704 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:05:13,705 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2476 [2023-09-07 17:05:13,706 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:05:13,706 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:05:13,707 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:05:13,707 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:05:13,708 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:05:13,717 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:05:13,719 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:05:13.719197 [2023-09-07 17:05:13,971 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:05:14,001 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:05:14,038 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:05:14,041 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:05:14,051 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:05:14,051 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694106313.9995272, 'te': 1694106314.0080156, 'initial_status': True} ====================================================================================================  [2023-09-07 17:05:14,053 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:05:14.053367 [2023-09-07 17:05:14,058 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:05:14,058 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:05:16,867 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106312, schain_finish_ts: 1693742304 [2023-09-07 17:05:18,871 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106312, schain_finish_ts: 1692637668 [2023-09-07 17:05:37,291 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:05:37,291 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:05:37,292 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:05:37,292 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:05:37,292 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:05:37,292 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:05:37,293 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:05:37,293 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:05:37,336 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:05:37,537 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:05:37,690 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:05:37,788 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:05:37,789 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2492 [2023-09-07 17:05:37,790 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:05:37,790 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:05:37,790 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:05:37,791 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:05:37,791 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:05:37,797 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:05:37,798 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:05:37.798938 [2023-09-07 17:05:37,937 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:05:37,963 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:05:37,997 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:05:37,999 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:05:38,006 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:05:38,007 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694106337.9616, 'te': 1694106337.9686332, 'initial_status': True} ====================================================================================================  [2023-09-07 17:05:38,009 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:05:38.009169 [2023-09-07 17:05:38,013 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:05:38,014 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:05:39,078 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:05:39,078 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:05:39,082 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:05:39.082817 [2023-09-07 17:05:39,089 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:05:39,089 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:05:39,099 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:05:39,104 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694106339.087956, 'te': 1694106339.1040664, 'initial_status': True} ====================================================================================================  [2023-09-07 17:05:39,106 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:05:39.106015 [2023-09-07 17:05:39,110 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:05:39,110 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:05:41,831 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:05:41,831 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:05:41,832 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:05:41,832 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:05:41,832 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:05:41,832 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:05:41,853 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:05:42,122 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:05:42,246 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:05:42,341 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:05:42,342 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2499 [2023-09-07 17:05:42,343 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:05:42,343 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:05:42,343 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:05:42,343 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:05:42,344 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:05:42,349 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:05:42,351 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:05:42.351567 [2023-09-07 17:05:42,482 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:05:42,506 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:05:42,534 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:05:42,536 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:05:42,543 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:05:42,543 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694106342.5048037, 'te': 1694106342.5106215, 'initial_status': True} ====================================================================================================  [2023-09-07 17:05:42,545 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:05:42.545342 [2023-09-07 17:05:42,550 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:05:42,550 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:05:44,397 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106336, schain_finish_ts: 1692377616 [2023-09-07 17:05:47,660 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:05:47,661 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:05:47,663 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:05:47.663013 [2023-09-07 17:05:47,669 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:05:47,669 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:05:47,675 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:05:47,680 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694106347.66807, 'te': 1694106347.680615, 'initial_status': True} ====================================================================================================  [2023-09-07 17:05:47,682 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:05:47.682714 [2023-09-07 17:05:47,689 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:05:47,689 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:05:52,107 ERROR][24912][submissive-well-off-rotanev][T_0] - core.schains.task:26 - Task submissive-well-off-rotanev-config failed with max() arg is an empty sequence Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 63, in execute self.am.upstream_config() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 194, in upstream_config new_config = create_new_upstream_config( File "/usr/src/admin/core/schains/config/main.py", line 51, in create_new_upstream_config schain_config = generate_schain_config_with_skale( File "/usr/src/admin/core/schains/config/generator.py", line 213, in generate_schain_config_with_skale node_groups = get_previous_schain_groups(skale, schain_name) File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 56, in get_previous_schain_groups _add_previous_schain_rotations_state( File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 119, in _add_previous_schain_rotations_state new_node_id = max(previous_nodes.items(), key=lambda x: x[1]['finish_ts'])[0] ValueError: max() arg is an empty sequence [2023-09-07 17:05:52,108 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:05:53,995 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106348, schain_finish_ts: 1693412520 [2023-09-07 17:05:55,918 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106348, schain_finish_ts: 0 [2023-09-07 17:05:57,668 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106348, schain_finish_ts: 1693742004 [2023-09-07 17:05:59,245 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106348, schain_finish_ts: 1692377364 [2023-09-07 17:06:05,340 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106360, schain_finish_ts: 0 [2023-09-07 17:06:09,586 INFO][1][MainProcess][MainThread] - core.schains.process_manager:117 - Got leaving sChains for the node: [] [2023-09-07 17:06:09,587 INFO][1][MainProcess][MainThread] - core.schains.process_manager:101 -  ==================================================================================================== Monitoring sChains Node ID: 16 sChains on node: [{'name': 'portly-passionate-sirius', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666281384, 'startBlock': 7803380, 'deposit': 0, 'index': 1, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x0000017723ca2', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'curly-red-alterf', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1676058936, 'startBlock': 8469646, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000001fe6cdfa', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'howling-hoedus', 'mainnetOwner': '0x1026467C91fdA40aCc7e6935d1E52f67274FD669', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666181400, 'startBlock': 7796660, 'deposit': 0, 'index': 0, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0xe5a600f4c2608', 'multitransactionMode': False, 'thresholdEncryption': False, 'active': True}, {'name': 'green-giddy-denebola', 'mainnetOwner': '0x8e835d5d9E28ffD259E8Fa8988Cacfc333A582B4', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674565164, 'startBlock': 8368049, 'deposit': 0, 'index': 4, 'generation': 1, 'originator': '0xd2001DAb6898127Be2F167B548691C87251D13C3', 'chainId': '0x00000585eb4b1', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'wan-tasty-kitalpha', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674568392, 'startBlock': 8368271, 'deposit': 0, 'index': 5, 'generation': 1, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x00000152e7be9', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'submissive-well-off-rotanev', 'mainnetOwner': '0x487360568fE5Ee4AE5c75782D5F63e9935B0E512', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1692361104, 'startBlock': 9539053, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000003791fa10', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}] Number of sChains on node: 6 Empty sChain structs: 1 ====================================================================================================  [2023-09-07 17:06:12,137 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:06:12,137 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:06:12,137 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:06:12,137 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:06:12,138 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:06:12,138 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:06:12,141 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:06:12,142 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:06:12,145 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:06:12,146 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:06:12,146 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:06:12,146 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:06:12,147 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:06:12,147 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:06:12,168 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:06:12,182 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:06:12,352 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:06:12,526 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:06:12,651 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:06:12,747 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:06:12,748 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2477 [2023-09-07 17:06:12,749 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:06:12,749 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:06:12,749 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:06:12,750 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:06:12,750 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:06:12,756 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:06:12,758 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:06:12.758219 [2023-09-07 17:06:12,797 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:06:12,909 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:06:12,911 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2491 [2023-09-07 17:06:12,912 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:06:12,912 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:06:12,914 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:06:12,914 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:06:12,915 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:06:12,921 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:06:12,923 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:06:12,923 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:06:12.923608 [2023-09-07 17:06:12,955 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:06:12,989 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:06:12,992 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:06:13,013 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:06:13,014 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694106372.9539144, 'te': 1694106372.9600692, 'initial_status': True} ====================================================================================================  [2023-09-07 17:06:13,018 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:06:13.018268 [2023-09-07 17:06:13,023 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:06:13,024 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:06:13,162 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:06:13,195 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:06:13,233 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:06:13,238 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:06:13,250 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:06:13,250 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694106373.1928673, 'te': 1694106373.1997895, 'initial_status': True} ====================================================================================================  [2023-09-07 17:06:13,255 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:06:13.255027 [2023-09-07 17:06:13,265 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:06:13,266 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:06:14,325 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:06:14,325 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:06:14,328 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:06:14.328578 [2023-09-07 17:06:14,336 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:06:14,337 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:06:14,346 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:06:14,351 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694106374.334041, 'te': 1694106374.3510344, 'initial_status': True} ====================================================================================================  [2023-09-07 17:06:14,353 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:06:14.353609 [2023-09-07 17:06:14,358 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:06:14,358 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:06:16,355 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:06:16,356 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain portly-passionate-sirius - Process is running: PID = 7 [2023-09-07 17:06:20,551 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:06:20,552 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:06:20,552 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:40 - Task submissive-well-off-rotanev-config finished with None [2023-09-07 17:06:20,552 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:42 - Running task submissive-well-off-rotanev-config [2023-09-07 17:06:20,552 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 17:06:22,990 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106372, schain_finish_ts: 1692637668 [2023-09-07 17:06:29,825 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:06:29,825 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:06:29,826 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:06:29,826 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:06:29,826 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:06:29,827 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:06:29,829 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:06:29,829 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 17:06:29,862 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:06:30,013 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:06:30,013 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain curly-red-alterf - Process is running: PID = 8 [2023-09-07 17:06:30,162 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:06:30,306 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:06:30,406 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:06:30,407 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2495 [2023-09-07 17:06:30,407 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:06:30,407 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:06:30,408 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:06:30,408 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:06:30,408 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:06:30,414 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:06:30,416 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:06:30.416359 [2023-09-07 17:06:30,548 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:06:30,575 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:06:30,605 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:06:30,607 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:06:30,614 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:06:30,615 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694106390.5739055, 'te': 1694106390.580078, 'initial_status': True} ====================================================================================================  [2023-09-07 17:06:30,616 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:06:30.616435 [2023-09-07 17:06:30,621 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:06:30,622 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:06:33,341 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:06:33,342 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain howling-hoedus - Process is running: PID = 9 [2023-09-07 17:06:39,625 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:06:39,625 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain green-giddy-denebola - Process is running: PID = 10 [2023-09-07 17:06:40,022 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:06:40,022 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:06:40,022 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:06:40,022 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:06:40,023 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:06:40,023 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:06:40,024 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:06:40,025 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 17:06:40,048 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:06:40,332 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:06:40,460 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:06:40,558 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:06:40,560 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2493 [2023-09-07 17:06:40,561 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:06:40,562 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:06:40,563 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:06:40,563 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:06:40,563 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:06:40,575 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:06:40,580 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:06:40.580180 [2023-09-07 17:06:40,756 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:06:40,783 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:06:40,815 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:06:40,817 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:06:40,824 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:06:40,824 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694106400.7817814, 'te': 1694106400.7885208, 'initial_status': True} ====================================================================================================  [2023-09-07 17:06:40,826 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:06:40.826187 [2023-09-07 17:06:40,830 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:06:40,831 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:06:41,483 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': False} [2023-09-07 17:06:41,483 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:06:41,487 INFO][24912][submissive-well-off-rotanev][T_0] - web.models.schain:152 - Changing monitor_last_seen for submissive-well-off-rotanev to 2023-09-07 17:06:41.486980 [2023-09-07 17:06:41,494 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.action:190 - Creating new upstream_config rotation_id: 8, stream: regression-2.2 [2023-09-07 17:06:41,496 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.config.main:49 - Generating sChain config for submissive-well-off-rotanev [2023-09-07 17:06:43,714 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:06:43,715 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:06:43,717 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:06:43.717961 [2023-09-07 17:06:43,726 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:06:43,727 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:06:43,733 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:06:43,738 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694106403.724989, 'te': 1694106403.7385387, 'initial_status': True} ====================================================================================================  [2023-09-07 17:06:43,740 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:06:43.740734 [2023-09-07 17:06:43,745 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:06:43,745 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:06:47,928 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106396, schain_finish_ts: 1693742304 [2023-09-07 17:06:49,490 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:06:49,491 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain wan-tasty-kitalpha - Process is running: PID = 11 [2023-09-07 17:06:58,054 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:06:58,055 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain submissive-well-off-rotanev - Process is running: PID = 24912 [2023-09-07 17:06:58,055 INFO][1][MainProcess][MainThread] - core.schains.process_manager:88 - Process manager procedure finished [2023-09-07 17:06:58,062 INFO][1][MainProcess][MainThread] - __main__:65 - Sleeping for 90s after run_process_manager [2023-09-07 17:07:00,774 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106408, schain_finish_ts: 1692377364 [2023-09-07 17:07:11,999 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106420, schain_finish_ts: 1693742004 [2023-09-07 17:07:17,105 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106432, schain_finish_ts: 0 [2023-09-07 17:07:20,848 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:07:20,848 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:07:20,850 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:07:20.850347 [2023-09-07 17:07:20,857 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:07:20,858 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:07:20,864 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:07:20,868 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694106440.855938, 'te': 1694106440.8681848, 'initial_status': True} ====================================================================================================  [2023-09-07 17:07:20,870 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:07:20.870326 [2023-09-07 17:07:20,874 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:07:20,875 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:07:25,814 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:07:25,814 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:07:25,816 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:07:25.816652 [2023-09-07 17:07:25,824 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:07:25,824 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:07:25,830 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:07:25,834 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694106445.822846, 'te': 1694106445.8345094, 'initial_status': True} ====================================================================================================  [2023-09-07 17:07:25,836 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:07:25.836153 [2023-09-07 17:07:25,840 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:07:25,840 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:07:28,687 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:07:28,691 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:07:28,694 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:07:28,694 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:07:28,695 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:07:28,696 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:07:28,720 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:07:28,959 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:07:29,108 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:07:29,234 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:07:29,237 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2500 [2023-09-07 17:07:29,238 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:07:29,239 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:07:29,240 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:07:29,241 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:07:29,241 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:07:29,256 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:07:29,256 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:07:29,257 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:07:29,257 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:07:29,257 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:07:29,257 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:07:29,259 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:07:29,259 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:07:29,263 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:07:29,269 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:07:29.269335 [2023-09-07 17:07:29,285 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:07:29,396 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106444, schain_finish_ts: 1693412520 [2023-09-07 17:07:29,502 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:07:29,541 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:07:29,573 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:07:29,577 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:07:29,586 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:07:29,587 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694106449.539739, 'te': 1694106449.5461335, 'initial_status': True} ====================================================================================================  [2023-09-07 17:07:29,589 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:07:29.589708 [2023-09-07 17:07:29,595 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:07:29,597 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:07:29,767 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:07:29,902 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:07:30,019 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:07:30,020 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2478 [2023-09-07 17:07:30,024 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:07:30,025 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:07:30,025 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:07:30,025 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:07:30,025 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:07:30,032 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:07:30,033 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:07:30.033970 [2023-09-07 17:07:30,171 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:07:30,209 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:07:30,253 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:07:30,255 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:07:30,262 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:07:30,263 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694106450.2077253, 'te': 1694106450.2157698, 'initial_status': True} ====================================================================================================  [2023-09-07 17:07:30,264 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:07:30.264975 [2023-09-07 17:07:30,270 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:07:30,271 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:07:30,752 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:07:30,753 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:07:30,753 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:07:30,753 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:07:30,754 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:07:30,754 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:07:30,774 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:07:31,034 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:07:31,238 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:07:31,348 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:07:31,350 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2492 [2023-09-07 17:07:31,351 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:07:31,351 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:07:31,351 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:07:31,352 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:07:31,352 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:07:31,359 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:07:31,361 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:07:31.361271 [2023-09-07 17:07:31,538 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:07:31,565 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:07:31,593 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:07:31,595 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:07:31,603 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:07:31,604 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694106451.5640903, 'te': 1694106451.570192, 'initial_status': True} ====================================================================================================  [2023-09-07 17:07:31,607 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:07:31.607081 [2023-09-07 17:07:31,612 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:07:31,614 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:07:37,332 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106444, schain_finish_ts: 1692637668 [2023-09-07 17:07:50,316 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:07:50,317 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:07:50,317 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 17:07:51,948 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 17:07:52,143 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:07:52,144 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:07:52,144 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:07:52,144 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:07:52,328 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:07:52,328 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:07:52,330 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 17:07:52,335 ERROR][11][wan-tasty-kitalpha][T_1] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:07:52,336 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:07:54,565 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:07:54,565 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:07:54,565 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:07:54,565 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:07:54,566 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:07:54,566 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:07:54,567 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:07:54,568 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:07:54,595 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:07:54,891 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:07:55,017 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:07:55,123 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:07:55,125 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2496 [2023-09-07 17:07:55,125 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:07:55,126 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:07:55,126 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:07:55,126 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:07:55,127 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:07:55,133 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:07:55,135 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:07:55.135914 [2023-09-07 17:07:55,310 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:07:55,350 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:07:55,403 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:07:55,406 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:07:55,415 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:07:55,416 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694106475.3477044, 'te': 1694106475.357319, 'initial_status': True} ====================================================================================================  [2023-09-07 17:07:55,419 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:07:55.418989 [2023-09-07 17:07:55,424 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:07:55,424 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:08:01,538 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:08:01,538 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:08:01,539 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:08:01,540 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:08:01,541 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:08:01,543 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:08:01,548 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:08:01,549 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:08:01,582 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:08:01,787 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:08:01,930 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:08:02,039 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:08:02,040 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2494 [2023-09-07 17:08:02,041 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:08:02,042 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:08:02,044 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:08:02,045 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:08:02,046 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:08:02,052 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:08:02,054 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:08:02.054336 [2023-09-07 17:08:02,440 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:08:02,466 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:08:02,495 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:08:02,497 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:08:02,504 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:08:02,504 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694106482.4643235, 'te': 1694106482.4716601, 'initial_status': True} ====================================================================================================  [2023-09-07 17:08:02,506 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:08:02.506572 [2023-09-07 17:08:02,511 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:08:02,511 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:08:04,145 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:08:04,145 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:08:04,148 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:08:04.148074 [2023-09-07 17:08:04,154 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:08:04,155 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:08:04,160 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:08:04,165 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694106484.1531253, 'te': 1694106484.1655986, 'initial_status': True} ====================================================================================================  [2023-09-07 17:08:04,167 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:08:04.167466 [2023-09-07 17:08:04,171 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:08:04,172 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:08:11,158 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106480, schain_finish_ts: 1693742304 [2023-09-07 17:08:14,796 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106492, schain_finish_ts: 1693742004 [2023-09-07 17:08:23,236 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106492, schain_finish_ts: 1692377364 [2023-09-07 17:08:23,302 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:08:23,304 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:08:23,309 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:08:23.309823 [2023-09-07 17:08:23,324 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:08:23,325 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:08:23,340 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:08:23,351 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694106503.3213434, 'te': 1694106503.3504422, 'initial_status': True} ====================================================================================================  [2023-09-07 17:08:23,358 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:08:23.358818 [2023-09-07 17:08:23,367 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:08:23,368 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:08:25,923 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:08:25,923 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:08:25,925 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:08:25.925644 [2023-09-07 17:08:25,932 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:08:25,933 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:08:25,939 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:08:25,944 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694106505.9311128, 'te': 1694106505.9437404, 'initial_status': True} ====================================================================================================  [2023-09-07 17:08:25,945 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:08:25.945583 [2023-09-07 17:08:25,951 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:08:25,951 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:08:27,142 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:08:27,142 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:08:27,142 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:08:27,143 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:08:27,143 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:08:27,143 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 17:08:27,168 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:08:27,365 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:08:27,494 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:08:27,596 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:08:27,597 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2493 [2023-09-07 17:08:27,597 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:08:27,598 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:08:27,598 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:08:27,598 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:08:27,598 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:08:27,605 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:08:27,607 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:08:27.607711 [2023-09-07 17:08:27,865 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:08:27,903 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:08:27,951 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:08:27,954 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:08:27,965 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:08:27,966 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694106507.9006133, 'te': 1694106507.9096684, 'initial_status': True} ====================================================================================================  [2023-09-07 17:08:27,969 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:08:27.969192 [2023-09-07 17:08:27,977 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:08:27,978 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:08:28,155 INFO][1][MainProcess][MainThread] - core.schains.cleaner:65 - Cleaner process started [2023-09-07 17:08:28,196 INFO][30603][cleaner][MainThread] - core.schains.cleaner:110 - Cleaner procedure started. [2023-09-07 17:08:33,203 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:08:33,204 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:08:33,204 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:08:33,204 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 17:08:33,205 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 17:08:33,205 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:08:33,212 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:08:33,216 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:08:33,248 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:08:33,657 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:08:33,844 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:08:33,949 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:08:33,950 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2501 [2023-09-07 17:08:33,950 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:08:33,951 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:08:33,951 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:08:33,951 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:08:33,952 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:08:33,958 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:08:33,959 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:08:33.959927 [2023-09-07 17:08:34,100 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:08:34,130 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:08:34,169 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:08:34,172 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:08:34,179 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:08:34,180 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694106514.1284285, 'te': 1694106514.137205, 'initial_status': True} ====================================================================================================  [2023-09-07 17:08:34,181 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:08:34.181745 [2023-09-07 17:08:34,186 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:08:34,187 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:08:44,469 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106504, schain_finish_ts: 1693412520 [2023-09-07 17:08:44,720 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:08:44,720 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:08:44,721 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:08:44,721 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:08:44,721 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:08:44,721 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:08:44,723 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:08:44,723 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 17:08:44,750 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:08:44,921 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:08:45,050 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:08:45,165 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:08:45,166 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2479 [2023-09-07 17:08:45,166 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:08:45,166 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:08:45,167 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:08:45,167 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:08:45,167 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:08:45,173 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:08:45,175 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:08:45.175362 [2023-09-07 17:08:45,307 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:08:45,332 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:08:45,360 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:08:45,363 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:08:45,370 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:08:45,371 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694106525.3311975, 'te': 1694106525.3383892, 'initial_status': True} ====================================================================================================  [2023-09-07 17:08:45,372 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:08:45.372727 [2023-09-07 17:08:45,376 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:08:45,377 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:08:48,946 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106504, schain_finish_ts: 0 [2023-09-07 17:08:53,612 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106528, schain_finish_ts: 1692637668 [2023-09-07 17:09:14,301 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:09:14,301 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:09:14,302 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:09:14,302 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:09:14,302 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:09:14,302 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:09:14,305 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:09:14,308 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 17:09:14,341 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:09:14,572 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:09:14,768 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:09:14,909 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:09:14,911 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2495 [2023-09-07 17:09:14,912 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:09:14,913 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:09:14,913 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:09:14,913 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:09:14,914 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:09:14,920 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:09:14,922 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:09:14.922721 [2023-09-07 17:09:15,068 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:09:15,095 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:09:15,127 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:09:15,130 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:09:15,139 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:09:15,140 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694106555.0932856, 'te': 1694106555.100266, 'initial_status': True} ====================================================================================================  [2023-09-07 17:09:15,142 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:09:15.142854 [2023-09-07 17:09:15,153 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:09:15,153 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:09:15,882 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:09:15,882 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:09:15,882 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:09:15,883 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:09:15,883 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:09:15,883 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:09:15,885 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:09:15,886 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:09:15,912 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:09:16,119 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:09:16,255 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:09:16,362 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:09:16,363 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2497 [2023-09-07 17:09:16,364 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:09:16,364 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:09:16,365 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:09:16,365 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:09:16,365 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:09:16,371 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:09:16,373 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:09:16.373846 [2023-09-07 17:09:16,512 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:09:16,537 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:09:16,565 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:09:16,568 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:09:16,575 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:09:16,575 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694106556.5357428, 'te': 1694106556.5418687, 'initial_status': True} ====================================================================================================  [2023-09-07 17:09:16,577 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:09:16.577117 [2023-09-07 17:09:16,581 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:09:16,582 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:09:17,060 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:09:17,062 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:09:17,065 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106552, schain_finish_ts: 1693742004 [2023-09-07 17:09:17,065 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 17:09:17,073 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106552, schain_finish_ts: 1693742304 [2023-09-07 17:09:17,122 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:09:17,122 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:09:17,124 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:09:17.124954 [2023-09-07 17:09:17,130 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:09:26,241 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:09:26,241 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:09:26,244 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:09:26.244124 [2023-09-07 17:09:26,252 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:09:26,253 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:09:26,260 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:09:26,264 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694106566.2503426, 'te': 1694106566.2646136, 'initial_status': True} ====================================================================================================  [2023-09-07 17:09:26,266 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:09:26.266820 [2023-09-07 17:09:26,271 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:09:26,272 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:09:37,005 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:09:37,005 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:09:37,006 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:09:37,006 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:09:37,006 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:09:37,006 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:09:37,033 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:09:37,214 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:09:37,363 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:09:37,480 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:09:37,481 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2502 [2023-09-07 17:09:37,482 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:09:37,482 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:09:37,483 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:09:37,483 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:09:37,483 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:09:37,489 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:09:37,491 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:09:37.491349 [2023-09-07 17:09:37,633 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:09:37,674 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:09:37,711 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:09:37,714 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:09:37,723 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:09:37,723 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694106577.672483, 'te': 1694106577.680858, 'initial_status': True} ====================================================================================================  [2023-09-07 17:09:37,725 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:09:37.725625 [2023-09-07 17:09:37,730 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:09:37,730 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:09:37,958 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106576, schain_finish_ts: 1692377364 [2023-09-07 17:09:45,970 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:09:45,970 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:09:45,971 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:09:45,971 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:09:45,971 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:09:45,972 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 17:09:45,995 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:09:46,308 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:09:46,441 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:09:46,556 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:09:46,557 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2494 [2023-09-07 17:09:46,558 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:09:46,558 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:09:46,558 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:09:46,559 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:09:46,559 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:09:46,566 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:09:46,567 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:09:46.567862 [2023-09-07 17:09:46,714 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:09:46,742 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:09:46,770 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:09:46,772 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:09:46,779 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:09:46,779 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694106586.7406259, 'te': 1694106586.7469149, 'initial_status': True} ====================================================================================================  [2023-09-07 17:09:46,781 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:09:46.781509 [2023-09-07 17:09:46,786 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:09:46,786 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:09:47,689 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:09:47,689 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:09:47,697 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:09:47.697642 [2023-09-07 17:09:47,709 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:09:47,709 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:09:47,718 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:09:47,723 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694106587.7066808, 'te': 1694106587.722927, 'initial_status': True} ====================================================================================================  [2023-09-07 17:09:47,725 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:09:47.725046 [2023-09-07 17:09:47,730 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:09:47,730 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:09:50,141 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:09:50,155 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:09:50,160 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:09:50.160857 [2023-09-07 17:09:50,171 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:09:50,172 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:09:50,193 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:09:50,200 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694106590.169933, 'te': 1694106590.2001584, 'initial_status': True} ====================================================================================================  [2023-09-07 17:09:50,204 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:09:50.204302 [2023-09-07 17:09:50,213 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:09:50,222 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:09:51,595 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:09:51,595 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:09:51,595 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:09:51,596 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:09:51,596 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:09:51,596 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:09:51,598 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:09:51,600 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 17:09:51,635 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:09:51,937 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:09:52,086 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:09:52,183 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:09:52,184 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2480 [2023-09-07 17:09:52,185 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:09:52,186 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:09:52,186 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:09:52,186 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:09:52,187 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:09:52,195 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:09:52,197 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:09:52.197408 [2023-09-07 17:09:52,340 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:09:52,369 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:09:52,408 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:09:52,410 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:09:52,420 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:09:52,420 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694106592.366644, 'te': 1694106592.3739824, 'initial_status': True} ====================================================================================================  [2023-09-07 17:09:52,422 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:09:52.422486 [2023-09-07 17:09:52,426 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:09:52,427 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:09:53,011 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106576, schain_finish_ts: 1692637668 [2023-09-07 17:09:54,709 INFO][30603][cleaner][MainThread] - core.schains.cleaner:116 - sChains on contracts: ['portly-passionate-sirius', 'curly-red-alterf', 'howling-hoedus', 'green-giddy-denebola', 'wan-tasty-kitalpha', '', 'submissive-well-off-rotanev'] sChains on node: ['curly-red-alterf', 'green-giddy-denebola', 'howling-hoedus', 'portly-passionate-sirius', 'submissive-well-off-rotanev', 'wan-tasty-kitalpha'] [2023-09-07 17:09:54,709 INFO][30603][cleaner][MainThread] - core.schains.cleaner:132 - Cleanup procedure finished [2023-09-07 17:09:54,714 INFO][1][MainProcess][MainThread] - core.schains.cleaner:67 - Cleaner process is joined. [2023-09-07 17:09:54,714 INFO][1][MainProcess][MainThread] - core.schains.cleaner:68 - Terminating the process [2023-09-07 17:09:54,715 INFO][1][MainProcess][MainThread] - __main__:70 - Sleeping for 90s after run_cleaner [2023-09-07 17:09:58,474 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:09:58,475 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:09:58,475 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:09:58,475 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:09:58,475 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:09:58,475 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:09:58,481 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:09:58,481 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 17:09:58,523 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:09:58,774 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:09:58,947 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:09:59,049 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:09:59,058 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2496 [2023-09-07 17:09:59,060 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:09:59,060 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:09:59,060 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:09:59,061 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:09:59,061 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:09:59,067 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:09:59,069 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:09:59.069452 [2023-09-07 17:09:59,332 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:09:59,361 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:09:59,388 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:09:59,390 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:09:59,399 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:09:59,399 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694106599.3593292, 'te': 1694106599.3656285, 'initial_status': True} ====================================================================================================  [2023-09-07 17:09:59,401 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:09:59.401445 [2023-09-07 17:09:59,406 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:09:59,407 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:10:00,627 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106576, schain_finish_ts: 1693412520 [2023-09-07 17:10:02,278 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106576, schain_finish_ts: 0 [2023-09-07 17:10:04,692 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:10:04,693 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:10:04,697 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:10:04.697742 [2023-09-07 17:10:04,712 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:10:04,713 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:10:04,722 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:10:04,727 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694106604.7098088, 'te': 1694106604.7270288, 'initial_status': True} ====================================================================================================  [2023-09-07 17:10:04,728 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:10:04.728927 [2023-09-07 17:10:04,734 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:10:04,734 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:10:04,912 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:10:04,912 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:10:04,912 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:10:04,912 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:10:04,913 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:10:04,913 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:10:04,916 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:10:04,916 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 17:10:04,950 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:10:05,196 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:10:05,198 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:10:05,207 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 17:10:05,274 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:10:05,448 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:10:05,554 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:10:05,555 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2498 [2023-09-07 17:10:05,556 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:10:05,556 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:10:05,556 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:10:05,557 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:10:05,557 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:10:05,565 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:10:05,567 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:10:05.567174 [2023-09-07 17:10:05,711 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:10:05,747 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:10:05,778 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:10:05,783 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:10:05,792 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:10:05,793 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694106605.7451725, 'te': 1694106605.7522092, 'initial_status': True} ====================================================================================================  [2023-09-07 17:10:05,794 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:10:05.794953 [2023-09-07 17:10:05,799 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:10:05,800 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:10:08,412 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:10:08,412 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:10:08,414 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:10:08.414977 [2023-09-07 17:10:08,423 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:10:08,423 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:10:08,429 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:10:08,434 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694106608.421633, 'te': 1694106608.4342356, 'initial_status': True} ====================================================================================================  [2023-09-07 17:10:08,437 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:10:08.436996 [2023-09-07 17:10:08,441 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:10:08,441 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:10:08,839 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 17:10:08,839 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 17:10:09,741 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:10:09,741 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:10:09,743 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:10:09.743801 [2023-09-07 17:10:09,757 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:10:09,758 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:10:09,766 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:10:09,772 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694106609.756538, 'te': 1694106609.7720478, 'initial_status': True} ====================================================================================================  [2023-09-07 17:10:09,773 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:10:09.773882 [2023-09-07 17:10:09,783 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:10:09,783 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:10:17,418 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106612, schain_finish_ts: 1693742304 [2023-09-07 17:10:19,302 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:10:19,303 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:10:19,303 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:10:19,303 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:10:19,303 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:10:19,304 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 17:10:19,331 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:10:19,675 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:10:19,869 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:10:19,989 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:10:19,990 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2503 [2023-09-07 17:10:19,990 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:10:19,991 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:10:19,991 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:10:19,991 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:10:19,992 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:10:19,997 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:10:19,999 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:10:19.999695 [2023-09-07 17:10:20,139 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:10:20,167 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:10:20,198 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:10:20,201 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:10:20,209 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:10:20,209 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694106620.1648953, 'te': 1694106620.1720388, 'initial_status': True} ====================================================================================================  [2023-09-07 17:10:20,212 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:10:20.212472 [2023-09-07 17:10:20,218 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:10:20,219 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:10:20,917 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106612, schain_finish_ts: 1693742004 [2023-09-07 17:10:23,554 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:10:23,554 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:10:23,555 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:10:23,555 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:10:23,555 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:10:23,555 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 17:10:23,578 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:10:23,818 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:10:24,086 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:10:24,200 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:10:24,201 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2495 [2023-09-07 17:10:24,201 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:10:24,202 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:10:24,202 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:10:24,204 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:10:24,205 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:10:24,219 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:10:24,223 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:10:24.223198 [2023-09-07 17:10:24,372 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:10:24,401 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:10:24,434 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:10:24,437 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:10:24,451 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:10:24,452 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694106624.3997192, 'te': 1694106624.4084408, 'initial_status': True} ====================================================================================================  [2023-09-07 17:10:24,453 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:10:24.453743 [2023-09-07 17:10:24,459 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:10:24,461 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:10:29,876 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106612, schain_finish_ts: 1692377364 [2023-09-07 17:10:32,835 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:10:32,835 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:10:32,835 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:10:32,836 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:10:32,836 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:10:32,836 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:10:32,839 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:10:32,842 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:10:32,867 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:10:33,115 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:10:33,293 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:10:33,397 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:10:33,398 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2481 [2023-09-07 17:10:33,398 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:10:33,399 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:10:33,399 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:10:33,399 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:10:33,400 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:10:33,406 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:10:33,408 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:10:33.408611 [2023-09-07 17:10:33,583 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:10:33,590 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106612, schain_finish_ts: 1693412520 [2023-09-07 17:10:33,610 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:10:33,645 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:10:33,648 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:10:33,659 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:10:33,660 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694106633.608979, 'te': 1694106633.6191347, 'initial_status': True} ====================================================================================================  [2023-09-07 17:10:33,661 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:10:33.661719 [2023-09-07 17:10:33,668 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:10:33,668 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:10:33,673 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106612, schain_finish_ts: 1692637668 [2023-09-07 17:10:39,995 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:10:39,996 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:10:39,996 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:10:39,996 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:10:39,996 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:10:39,997 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:10:39,998 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:10:39,998 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 17:10:40,039 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:10:40,308 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:10:40,475 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 17:10:40,475 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 17:10:40,510 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:10:40,628 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:10:40,629 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:10:40,631 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:10:40.631309 [2023-09-07 17:10:40,647 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:10:40,648 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:10:40,658 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:10:40,663 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:10:40,664 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2497 [2023-09-07 17:10:40,665 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:10:40,666 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:10:40,666 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:10:40,666 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:10:40,667 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:10:40,667 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694106640.6442335, 'te': 1694106640.6672196, 'initial_status': True} ====================================================================================================  [2023-09-07 17:10:40,669 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:10:40.669349 [2023-09-07 17:10:40,675 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:10:40,676 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:10:40,676 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:10:40,679 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:10:40.679282 [2023-09-07 17:10:40,861 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:10:40,901 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:10:40,947 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:10:40,948 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:10:40,949 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:10:40,950 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:10:40,950 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:10:40,951 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:10:40,953 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:10:40,957 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 17:10:40,965 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:10:40,969 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:10:40,980 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:10:40,981 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694106640.8905354, 'te': 1694106640.9091754, 'initial_status': True} ====================================================================================================  [2023-09-07 17:10:40,983 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:10:40.983662 [2023-09-07 17:10:40,987 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:10:40,991 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:10:40,993 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:10:41,211 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:10:41,396 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:10:41,528 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:10:41,529 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2499 [2023-09-07 17:10:41,530 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:10:41,530 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:10:41,530 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:10:41,531 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:10:41,531 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:10:41,538 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:10:41,540 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:10:41.540251 [2023-09-07 17:10:41,813 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:10:41,847 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:10:41,891 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:10:41,894 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:10:41,908 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:10:41,909 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694106641.8454895, 'te': 1694106641.8550937, 'initial_status': True} ====================================================================================================  [2023-09-07 17:10:41,911 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:10:41.911831 [2023-09-07 17:10:41,922 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:10:41,923 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:10:42,185 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106636, schain_finish_ts: 0 [2023-09-07 17:10:42,374 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:10:42,374 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:10:42,376 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:10:42.376478 [2023-09-07 17:10:42,384 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:10:42,384 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:10:42,392 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:10:42,398 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694106642.3830814, 'te': 1694106642.3984692, 'initial_status': True} ====================================================================================================  [2023-09-07 17:10:42,401 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:10:42.401613 [2023-09-07 17:10:42,408 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:10:42,408 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:10:43,104 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:10:43,105 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:10:43,107 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:10:43.107961 [2023-09-07 17:10:43,117 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:10:43,117 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:10:43,124 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:10:43,129 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694106643.1149464, 'te': 1694106643.1296525, 'initial_status': True} ====================================================================================================  [2023-09-07 17:10:43,132 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:10:43.132293 [2023-09-07 17:10:43,137 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:10:43,138 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:10:45,043 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:10:45,043 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:10:45,043 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 17:10:47,641 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 17:10:47,827 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:10:47,828 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:10:47,828 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:10:47,828 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:10:47,992 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:10:47,992 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:10:47,994 INFO][8][curly-red-alterf][T_0] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 17:10:47,999 ERROR][8][curly-red-alterf][T_0] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:10:48,000 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:10:57,792 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106648, schain_finish_ts: 1692377364 [2023-09-07 17:10:58,947 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:10:58,947 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:10:58,947 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:10:58,948 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:10:58,948 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:10:58,948 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:10:58,949 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:10:58,950 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:10:58,977 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:10:59,303 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:10:59,446 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:10:59,552 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:10:59,553 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2482 [2023-09-07 17:10:59,554 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:10:59,554 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:10:59,555 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:10:59,555 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:10:59,555 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:10:59,561 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:10:59,563 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:10:59.563405 [2023-09-07 17:10:59,742 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:10:59,771 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:10:59,807 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:10:59,809 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:10:59,817 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:10:59,820 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694106659.768502, 'te': 1694106659.7788134, 'initial_status': True} ====================================================================================================  [2023-09-07 17:10:59,823 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:10:59.823494 [2023-09-07 17:10:59,828 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:10:59,829 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:11:00,310 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106648, schain_finish_ts: 1693742304 [2023-09-07 17:11:00,732 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:11:00,733 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:11:00,735 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:11:00.735177 [2023-09-07 17:11:00,742 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:11:00,743 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:11:00,749 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:11:00,754 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694106660.7414863, 'te': 1694106660.7545822, 'initial_status': True} ====================================================================================================  [2023-09-07 17:11:00,756 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:11:00.756343 [2023-09-07 17:11:00,761 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:11:00,761 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:11:01,466 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:11:01,466 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:11:01,467 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:11:01,467 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:11:01,467 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:11:01,467 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:11:01,491 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:11:01,711 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:11:01,870 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:11:01,976 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:11:01,977 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2504 [2023-09-07 17:11:01,978 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:11:01,978 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:11:01,979 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:11:01,979 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:11:01,979 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:11:01,986 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:11:01,989 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:11:01.989066 [2023-09-07 17:11:02,173 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:11:02,207 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:11:02,237 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 17:11:02,243 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:11:02,246 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:11:02,254 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:11:02,255 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694106662.2063274, 'te': 1694106662.21429, 'initial_status': True} ====================================================================================================  [2023-09-07 17:11:02,257 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:11:02.257440 [2023-09-07 17:11:02,262 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:11:02,263 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:11:02,412 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:11:02,412 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:11:02,412 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:11:02,413 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:11:02,591 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:11:02,591 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:11:02,594 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 17:11:02,601 ERROR][11][wan-tasty-kitalpha][T_0] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:11:02,601 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:11:04,524 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106660, schain_finish_ts: 1693742004 [2023-09-07 17:11:05,695 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:11:05,696 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:11:05,696 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:11:05,696 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 17:11:05,696 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 17:11:05,697 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:11:05,697 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:11:05,699 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:11:05,732 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:11:05,965 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:11:06,012 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106660, schain_finish_ts: 0 [2023-09-07 17:11:06,126 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:11:06,238 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:11:06,239 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2496 [2023-09-07 17:11:06,240 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:11:06,240 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:11:06,240 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:11:06,241 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:11:06,241 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:11:06,252 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:11:06,255 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:11:06.255569 [2023-09-07 17:11:06,440 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:11:06,501 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:11:06,544 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:11:06,546 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:11:06,560 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:11:06,561 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694106666.4976697, 'te': 1694106666.507631, 'initial_status': True} ====================================================================================================  [2023-09-07 17:11:06,565 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:11:06.564933 [2023-09-07 17:11:06,570 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:11:06,572 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:11:07,202 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:11:07,203 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:11:07,203 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 17:11:07,505 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:11:07,505 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:11:07,512 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:11:07.511271 [2023-09-07 17:11:07,522 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:11:10,974 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106660, schain_finish_ts: 1692637668 [2023-09-07 17:11:11,928 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106660, schain_finish_ts: 1693412520 [2023-09-07 17:11:12,122 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:11:12,123 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:11:12,123 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:11:12,123 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:11:12,129 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:11:12,145 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:11:12,147 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:11:12,147 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:11:12,202 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:11:12,425 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:11:12,694 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:11:12,799 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:11:12,801 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2498 [2023-09-07 17:11:12,801 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:11:12,802 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:11:12,802 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:11:12,802 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:11:12,802 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:11:12,808 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:11:12,810 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:11:12.810336 [2023-09-07 17:11:12,977 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:11:13,002 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:11:13,039 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:11:13,041 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:11:13,048 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:11:13,048 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694106673.001003, 'te': 1694106673.00847, 'initial_status': True} ====================================================================================================  [2023-09-07 17:11:13,049 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:11:13.049898 [2023-09-07 17:11:13,055 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:11:13,056 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:11:13,108 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:11:13,108 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:11:13,108 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:11:13,109 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:11:13,109 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:11:13,109 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:11:13,112 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:11:13,112 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 17:11:13,140 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:11:13,352 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:11:13,528 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:11:13,642 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:11:13,644 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2500 [2023-09-07 17:11:13,644 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:11:13,645 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:11:13,645 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:11:13,645 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:11:13,646 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:11:13,652 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:11:13,654 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:11:13.654656 [2023-09-07 17:11:13,829 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:11:13,858 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:11:13,889 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:11:13,891 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:11:13,899 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:11:13,900 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694106673.8564394, 'te': 1694106673.8633008, 'initial_status': True} ====================================================================================================  [2023-09-07 17:11:13,905 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:11:13.905335 [2023-09-07 17:11:13,913 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:11:13,913 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:11:13,939 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:11:13,939 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:11:13,941 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:11:13.941867 [2023-09-07 17:11:13,948 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:11:13,949 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:11:13,957 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:11:13,963 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694106673.9473972, 'te': 1694106673.9628026, 'initial_status': True} ====================================================================================================  [2023-09-07 17:11:13,965 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:11:13.965031 [2023-09-07 17:11:13,969 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:11:13,969 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:11:14,870 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:11:14,870 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:11:14,873 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:11:14.873576 [2023-09-07 17:11:14,888 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:11:14,888 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:11:14,897 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:11:14,907 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694106674.8864005, 'te': 1694106674.9067144, 'initial_status': True} ====================================================================================================  [2023-09-07 17:11:14,909 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:11:14.909594 [2023-09-07 17:11:14,917 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:11:14,917 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:11:21,927 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106672, schain_finish_ts: 1692377364 [2023-09-07 17:11:23,099 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:11:23,099 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:11:23,099 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:11:23,100 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:11:23,100 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:11:23,100 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:11:23,101 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:11:23,104 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 17:11:23,130 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:11:23,334 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:11:23,507 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:11:23,648 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:11:23,658 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2483 [2023-09-07 17:11:23,662 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:11:23,663 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:11:23,666 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:11:23,669 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:11:23,669 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:11:23,677 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:11:23,679 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:11:23.679921 [2023-09-07 17:11:23,840 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:11:23,867 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:11:23,897 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:11:23,900 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:11:23,907 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:11:23,908 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694106683.8652809, 'te': 1694106683.8725512, 'initial_status': True} ====================================================================================================  [2023-09-07 17:11:23,909 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:11:23.909929 [2023-09-07 17:11:23,914 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:11:23,915 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:11:24,805 INFO][1][MainProcess][MainThread] - core.schains.process_manager:57 - Process manager started [2023-09-07 17:11:24,847 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:11:24,847 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:11:24,849 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:11:24.849741 [2023-09-07 17:11:24,859 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:11:24,859 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:11:24,865 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:11:24,871 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694106684.857757, 'te': 1694106684.8715737, 'initial_status': True} ====================================================================================================  [2023-09-07 17:11:24,873 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:11:24.873537 [2023-09-07 17:11:24,881 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:11:24,881 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:11:25,021 INFO][1][MainProcess][MainThread] - core.schains.notifications:35 - Node account has 406869999689179237 WEI [2023-09-07 17:11:25,022 INFO][1][MainProcess][MainThread] - tools.notifications.messages:188 - Saving new balance state 3009 1 [2023-09-07 17:11:25,023 INFO][1][MainProcess][MainThread] - core.schains.process_manager:95 - Fetching schains to monitor... [2023-09-07 17:11:26,862 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 17:11:26,862 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 17:11:28,157 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106684, schain_finish_ts: 0 [2023-09-07 17:11:28,641 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106684, schain_finish_ts: 1693742004 [2023-09-07 17:11:29,344 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:11:29,344 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:11:29,345 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 17:11:29,774 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:11:29,774 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:11:29,775 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:11:29,775 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:11:29,775 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:11:29,775 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 17:11:29,795 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:11:30,171 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:11:30,331 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:11:30,393 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106684, schain_finish_ts: 1693742304 [2023-09-07 17:11:30,448 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:11:30,450 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2497 [2023-09-07 17:11:30,450 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:11:30,451 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:11:30,451 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:11:30,451 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:11:30,452 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:11:30,458 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:11:30,460 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:11:30.460590 [2023-09-07 17:11:30,627 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:11:30,652 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:11:30,680 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:11:30,682 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:11:30,687 INFO][1][MainProcess][MainThread] - core.schains.process_manager:109 - Get leaving_history for node ... [2023-09-07 17:11:30,690 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:11:30,691 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694106690.6506987, 'te': 1694106690.6577759, 'initial_status': True} ====================================================================================================  [2023-09-07 17:11:30,692 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:11:30.692880 [2023-09-07 17:11:30,698 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:11:30,699 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:11:31,520 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:11:31,520 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:11:31,520 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:11:31,521 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 17:11:31,521 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 17:11:31,521 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:11:31,522 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:11:31,523 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:11:31,550 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:11:31,904 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:11:32,071 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:11:32,184 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:11:32,185 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2505 [2023-09-07 17:11:32,185 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:11:32,186 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:11:32,186 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:11:32,186 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:11:32,186 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:11:32,192 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:11:32,194 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:11:32.194813 [2023-09-07 17:11:32,380 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:11:32,423 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:11:32,464 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:11:32,467 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:11:32,478 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:11:32,478 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694106692.4210851, 'te': 1694106692.4283686, 'initial_status': True} ====================================================================================================  [2023-09-07 17:11:32,480 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:11:32.480330 [2023-09-07 17:11:32,484 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:11:32,485 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:11:32,677 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106684, schain_finish_ts: 1692377616 [2023-09-07 17:11:33,373 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:11:33,373 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:11:33,376 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:11:33.376215 [2023-09-07 17:11:33,386 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:11:34,661 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106684, schain_finish_ts: 0 [2023-09-07 17:11:35,030 INFO][1][MainProcess][MainThread] - core.schains.process_manager:117 - Got leaving sChains for the node: [] [2023-09-07 17:11:35,031 INFO][1][MainProcess][MainThread] - core.schains.process_manager:101 -  ==================================================================================================== Monitoring sChains Node ID: 16 sChains on node: [{'name': 'portly-passionate-sirius', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666281384, 'startBlock': 7803380, 'deposit': 0, 'index': 1, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x0000017723ca2', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'curly-red-alterf', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1676058936, 'startBlock': 8469646, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000001fe6cdfa', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'howling-hoedus', 'mainnetOwner': '0x1026467C91fdA40aCc7e6935d1E52f67274FD669', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666181400, 'startBlock': 7796660, 'deposit': 0, 'index': 0, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0xe5a600f4c2608', 'multitransactionMode': False, 'thresholdEncryption': False, 'active': True}, {'name': 'green-giddy-denebola', 'mainnetOwner': '0x8e835d5d9E28ffD259E8Fa8988Cacfc333A582B4', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674565164, 'startBlock': 8368049, 'deposit': 0, 'index': 4, 'generation': 1, 'originator': '0xd2001DAb6898127Be2F167B548691C87251D13C3', 'chainId': '0x00000585eb4b1', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'wan-tasty-kitalpha', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674568392, 'startBlock': 8368271, 'deposit': 0, 'index': 5, 'generation': 1, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x00000152e7be9', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'free-basic-rosy-SKALE', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1693996920, 'startBlock': 9646443, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000002bb90439', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'submissive-well-off-rotanev', 'mainnetOwner': '0x487360568fE5Ee4AE5c75782D5F63e9935B0E512', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1692361104, 'startBlock': 9539053, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000003791fa10', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}] Number of sChains on node: 7 Empty sChain structs: 0 ====================================================================================================  [2023-09-07 17:11:35,410 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:11:35,410 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain portly-passionate-sirius - Process is running: PID = 7 [2023-09-07 17:11:35,783 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:11:35,783 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain curly-red-alterf - Process is running: PID = 8 [2023-09-07 17:11:36,156 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:11:36,156 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain howling-hoedus - Process is running: PID = 9 [2023-09-07 17:11:36,527 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:11:36,527 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain green-giddy-denebola - Process is running: PID = 10 [2023-09-07 17:11:36,909 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:11:36,910 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain wan-tasty-kitalpha - Process is running: PID = 11 [2023-09-07 17:11:37,304 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:11:37,304 INFO][1][MainProcess][MainThread] - core.schains.process_manager:72 - sChain free-basic-rosy-SKALE - PID 0 is not running, spawning... [2023-09-07 17:11:37,309 INFO][1][MainProcess][MainThread] - web.models.schain:157 - Changing monitor_id for free-basic-rosy-SKALE to 30622 [2023-09-07 17:11:37,317 INFO][1][MainProcess][MainThread] - core.schains.process_manager:85 - sChain free-basic-rosy-SKALE - Process started: PID = 30622 [2023-09-07 17:11:37,703 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:11:37,704 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain submissive-well-off-rotanev - Process is running: PID = 24912 [2023-09-07 17:11:37,704 INFO][1][MainProcess][MainThread] - core.schains.process_manager:88 - Process manager procedure finished [2023-09-07 17:11:37,704 INFO][1][MainProcess][MainThread] - __main__:65 - Sleeping for 90s after run_process_manager [2023-09-07 17:11:38,257 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106696, schain_finish_ts: 1694108304 [2023-09-07 17:11:39,388 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:11:39,388 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:11:39,389 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.task:42 - Running task free-basic-rosy-SKALE-config [2023-09-07 17:11:39,399 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:11:41,164 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': False, 'dkg': False, 'external_state': False, 'upstream_config': False} [2023-09-07 17:11:41,165 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:11:41,168 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:152 - Changing monitor_last_seen for free-basic-rosy-SKALE to 2023-09-07 17:11:41.168106 [2023-09-07 17:11:41,175 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.action:160 - Initializing config dir [2023-09-07 17:11:41,175 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:11:44,044 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106696, schain_finish_ts: 1693412520 [2023-09-07 17:11:45,194 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:11:45,194 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:11:45,194 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:11:45,195 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:11:45,195 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:11:45,195 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:11:45,195 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:11:45,196 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 17:11:45,232 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:11:45,483 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:11:45,827 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:11:45,969 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:11:45,971 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2501 [2023-09-07 17:11:45,971 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:11:45,972 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:11:45,972 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:11:45,972 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:11:45,973 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:11:45,983 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:11:45,985 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:11:45.985902 [2023-09-07 17:11:46,249 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:11:46,292 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:11:46,337 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:11:46,340 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:11:46,358 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:11:46,358 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694106706.2839675, 'te': 1694106706.298216, 'initial_status': True} ====================================================================================================  [2023-09-07 17:11:46,360 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:11:46.360390 [2023-09-07 17:11:46,365 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:11:46,365 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:11:46,498 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 17:11:46,681 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:11:46,682 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:11:46,682 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:11:46,683 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:11:46,849 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:11:46,850 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:11:46,851 INFO][8][curly-red-alterf][T_1] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 17:11:46,858 ERROR][8][curly-red-alterf][T_1] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:11:46,858 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:11:46,985 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:11:46,986 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:11:46,988 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:11:46.988213 [2023-09-07 17:11:46,997 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:11:46,997 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:11:47,003 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:11:47,007 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694106706.9956915, 'te': 1694106707.0070565, 'initial_status': True} ====================================================================================================  [2023-09-07 17:11:47,009 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:11:47.009127 [2023-09-07 17:11:47,013 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:11:47,013 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:11:52,101 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106708, schain_finish_ts: 1692637668 [2023-09-07 17:11:55,424 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 17:11:55,425 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 17:12:00,388 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106708, schain_finish_ts: 1692377364 [2023-09-07 17:12:00,482 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:12:00,482 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:12:00,482 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:12:00,483 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:12:00,483 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:12:00,483 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:12:00,485 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:12:00,488 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 17:12:00,524 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:12:00,819 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:12:01,008 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:12:01,134 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:12:01,135 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2499 [2023-09-07 17:12:01,136 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:12:01,136 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:12:01,137 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:12:01,137 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:12:01,137 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:12:01,147 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:12:01,150 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:12:01.150947 [2023-09-07 17:12:01,354 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:12:01,384 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:12:01,414 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:12:01,417 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:12:01,427 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:12:01,427 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694106721.381457, 'te': 1694106721.389658, 'initial_status': True} ====================================================================================================  [2023-09-07 17:12:01,429 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:12:01.429902 [2023-09-07 17:12:01,436 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:12:01,437 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:12:02,288 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:12:02,289 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:12:02,289 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:12:02,289 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:12:02,289 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:12:02,290 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:12:02,295 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:12:02,297 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 17:12:02,321 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:12:02,638 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:12:02,769 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:12:02,880 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:12:02,881 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2484 [2023-09-07 17:12:02,882 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:12:02,882 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:12:02,882 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:12:02,882 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:12:02,883 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:12:02,889 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:12:02,891 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:12:02.891862 [2023-09-07 17:12:03,033 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:12:03,060 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:12:03,102 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:12:03,104 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:12:03,112 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:12:03,115 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694106723.0586703, 'te': 1694106723.065846, 'initial_status': True} ====================================================================================================  [2023-09-07 17:12:03,117 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:12:03.117116 [2023-09-07 17:12:03,128 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:12:03,129 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:12:04,901 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:12:04,901 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:12:04,903 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:12:04.903750 [2023-09-07 17:12:04,915 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:12:04,915 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:12:04,920 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:12:04,925 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694106724.9135044, 'te': 1694106724.925085, 'initial_status': True} ====================================================================================================  [2023-09-07 17:12:04,926 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:12:04.926923 [2023-09-07 17:12:04,931 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:12:04,932 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:12:08,714 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106720, schain_finish_ts: 0 [2023-09-07 17:12:10,859 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106720, schain_finish_ts: 1693412520 [2023-09-07 17:12:12,088 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:12:12,088 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:12:12,093 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:12:12.093387 [2023-09-07 17:12:12,106 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:12:12,107 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:12:12,119 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:12:12,127 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694106732.101603, 'te': 1694106732.126269, 'initial_status': True} ====================================================================================================  [2023-09-07 17:12:12,132 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:12:12.132278 [2023-09-07 17:12:12,140 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:12:12,141 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:12:13,147 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:12:13,148 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:12:13,149 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:12:13,915 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106720, schain_finish_ts: 1693742004 [2023-09-07 17:12:13,923 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106720, schain_finish_ts: 1693742304 [2023-09-07 17:12:15,850 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:12:15,850 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:12:15,850 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:12:15,850 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:12:15,851 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:12:15,851 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:12:15,853 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:12:15,856 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:12:15,883 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:12:16,101 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:12:16,270 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:12:16,375 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:12:16,376 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2502 [2023-09-07 17:12:16,377 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:12:16,377 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:12:16,379 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:12:16,381 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:12:16,382 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:12:16,396 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:12:16,401 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:12:16.401389 [2023-09-07 17:12:16,543 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:12:16,568 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:12:16,595 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:12:16,596 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:12:16,603 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:12:16,603 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694106736.5666869, 'te': 1694106736.572578, 'initial_status': True} ====================================================================================================  [2023-09-07 17:12:16,605 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:12:16.605376 [2023-09-07 17:12:16,610 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:12:16,611 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:12:17,845 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:12:17,845 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:12:17,845 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:12:17,846 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 17:12:17,846 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 17:12:17,846 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:12:17,848 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:12:17,850 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:12:17,871 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:12:18,004 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:12:18,005 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:12:18,005 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:12:18,005 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:12:18,005 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:12:18,006 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:12:18,026 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:12:18,078 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:12:18,209 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:12:18,307 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:12:18,308 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2498 [2023-09-07 17:12:18,309 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:12:18,309 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:12:18,309 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:12:18,309 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:12:18,310 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:12:18,315 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:12:18,317 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:12:18.317730 [2023-09-07 17:12:18,352 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:12:18,490 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:12:18,515 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:12:18,550 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:12:18,552 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:12:18,611 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:12:18,611 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694106738.5139155, 'te': 1694106738.5206244, 'initial_status': True} ====================================================================================================  [2023-09-07 17:12:18,613 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:12:18.613506 [2023-09-07 17:12:18,618 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:12:18,619 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:12:18,655 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:12:18,764 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:12:18,765 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2506 [2023-09-07 17:12:18,765 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:12:18,766 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:12:18,766 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:12:18,766 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:12:18,766 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:12:18,773 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:12:18,774 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:12:18.774885 [2023-09-07 17:12:19,026 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:12:19,057 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:12:19,088 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:12:19,090 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:12:19,098 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:12:19,098 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694106739.054852, 'te': 1694106739.0641453, 'initial_status': True} ====================================================================================================  [2023-09-07 17:12:19,100 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:12:19.100432 [2023-09-07 17:12:19,106 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:12:19,107 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:12:21,392 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106720, schain_finish_ts: 1694108304 [2023-09-07 17:12:22,079 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:12:22,079 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:12:22,081 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:12:22.081288 [2023-09-07 17:12:22,088 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:12:22,089 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:12:22,095 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:12:22,100 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694106742.0872624, 'te': 1694106742.100156, 'initial_status': True} ====================================================================================================  [2023-09-07 17:12:22,102 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:12:22.102566 [2023-09-07 17:12:22,107 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:12:22,108 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:12:23,144 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:12:23,144 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:12:23,146 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:12:23.146574 [2023-09-07 17:12:23,151 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:12:25,315 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:12:25,315 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:12:25,315 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 17:12:26,582 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:159 - sChain: free-basic-rosy-SKALE. Node id on chain is 14; Node id on contract is 16 [2023-09-07 17:12:26,582 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:162 - sChain: free-basic-rosy-SKALE. DKG timeout is 1800 [2023-09-07 17:12:29,705 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106744, schain_finish_ts: 1692637668 [2023-09-07 17:12:33,605 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:12:33,605 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:12:33,605 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:12:33,605 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:12:33,606 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:12:33,606 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:12:33,607 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:12:33,609 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:12:33,634 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:12:33,847 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:12:34,118 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:12:34,226 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:12:34,227 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2500 [2023-09-07 17:12:34,228 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:12:34,228 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:12:34,228 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:12:34,228 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:12:34,229 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:12:34,235 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:12:34,241 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:12:34.241131 [2023-09-07 17:12:34,407 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:12:34,440 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:12:34,474 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:12:34,477 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:12:34,486 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:12:34,486 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694106754.4385774, 'te': 1694106754.4455771, 'initial_status': True} ====================================================================================================  [2023-09-07 17:12:34,487 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:12:34.487785 [2023-09-07 17:12:34,492 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:12:34,493 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:12:39,391 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106744, schain_finish_ts: 1692377364 [2023-09-07 17:12:39,570 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:12:39,571 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:12:39,574 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:12:39.574281 [2023-09-07 17:12:39,583 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:12:39,584 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:12:39,590 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:12:39,596 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694106759.5814342, 'te': 1694106759.595939, 'initial_status': True} ====================================================================================================  [2023-09-07 17:12:39,598 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:12:39.598571 [2023-09-07 17:12:39,603 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:12:39,603 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:12:43,172 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:12:43,173 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:12:43,176 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:12:43,177 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:12:43,179 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:12:43,179 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:12:43,184 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:12:43,184 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 17:12:43,217 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:12:43,437 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:12:43,592 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:12:43,726 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:12:43,727 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2485 [2023-09-07 17:12:43,728 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:12:43,728 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:12:43,728 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:12:43,728 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:12:43,729 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:12:43,735 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:12:43,737 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:12:43.737106 [2023-09-07 17:12:43,956 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:12:43,985 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:12:44,023 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:12:44,029 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:12:44,041 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:12:44,042 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694106763.9822915, 'te': 1694106763.9898095, 'initial_status': True} ====================================================================================================  [2023-09-07 17:12:44,046 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:12:44.046596 [2023-09-07 17:12:44,052 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:12:44,054 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:12:46,149 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:12:46,150 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:12:46,153 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:12:46.153116 [2023-09-07 17:12:46,161 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:12:46,161 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:12:46,167 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:12:46,173 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694106766.1596937, 'te': 1694106766.1734536, 'initial_status': True} ====================================================================================================  [2023-09-07 17:12:46,176 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:12:46.176827 [2023-09-07 17:12:46,181 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:12:46,182 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:12:49,766 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:40 - Collecting rotation history for submissive-well-off-rotanev... [2023-09-07 17:12:51,376 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106744, schain_finish_ts: 1693742304 [2023-09-07 17:12:56,128 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106744, schain_finish_ts: 0 [2023-09-07 17:12:56,380 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:12:56,381 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:12:56,381 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:12:56,381 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:12:56,381 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:12:56,381 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 17:12:56,404 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:12:56,585 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:12:56,711 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:12:56,807 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:12:56,808 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2507 [2023-09-07 17:12:56,809 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:12:56,809 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:12:56,809 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:12:56,809 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:12:56,810 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:12:56,816 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:12:56,817 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:12:56.817808 [2023-09-07 17:12:56,947 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:12:56,973 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:12:57,000 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:12:57,001 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:12:57,011 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:12:57,012 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694106776.971895, 'te': 1694106776.977939, 'initial_status': True} ====================================================================================================  [2023-09-07 17:12:57,013 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:12:57.013895 [2023-09-07 17:12:57,017 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:12:57,018 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:12:57,564 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:50 - Rotation data for submissive-well-off-rotanev: Rotation(leaving_node_id=65, new_node_id=65, freeze_until=1694108100, rotation_counter=8) [2023-09-07 17:12:57,576 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106744, schain_finish_ts: 1693742004 [2023-09-07 17:12:57,672 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106744, schain_finish_ts: 1693412520 [2023-09-07 17:13:01,183 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:13:01,183 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:13:01,183 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:13:01,184 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:13:01,184 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:13:01,184 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:13:01,184 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:13:01,188 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 17:13:01,221 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:13:01,426 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:13:01,427 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:13:01,427 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 17:13:01,628 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:13:01,797 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:13:01,862 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106744, schain_finish_ts: 1694108304 [2023-09-07 17:13:01,902 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:13:01,903 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2503 [2023-09-07 17:13:01,903 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:13:01,904 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:13:01,904 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:13:01,904 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:13:01,905 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:13:01,911 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:13:01,913 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:13:01.913084 [2023-09-07 17:13:01,923 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:13:01,924 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:13:01,924 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:13:01,924 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:13:01,924 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:13:01,925 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 17:13:01,948 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:13:02,101 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:13:02,138 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:13:02,176 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:13:02,179 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:13:02,188 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:13:02,188 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694106782.1361413, 'te': 1694106782.1424177, 'initial_status': True} ====================================================================================================  [2023-09-07 17:13:02,190 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:13:02.190706 [2023-09-07 17:13:02,194 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:13:02,196 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:13:02,365 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:13:02,506 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:13:02,621 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:13:02,624 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2499 [2023-09-07 17:13:02,626 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:13:02,627 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:13:02,628 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:13:02,628 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:13:02,631 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:13:02,639 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:13:02,644 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:13:02.644293 [2023-09-07 17:13:02,848 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:13:02,881 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:13:02,912 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:13:02,914 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:13:02,922 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:13:02,923 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694106782.879155, 'te': 1694106782.8866434, 'initial_status': True} ====================================================================================================  [2023-09-07 17:13:02,928 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:13:02.928771 [2023-09-07 17:13:02,935 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:13:02,936 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:13:07,186 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 17:13:07,351 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:13:07,351 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:13:07,352 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:13:07,358 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:13:07,358 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:13:07,358 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:13:07,358 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:13:07,532 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:13:07,532 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:13:07,534 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 17:13:07,540 ERROR][11][wan-tasty-kitalpha][T_1] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:13:07,540 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:13:07,725 INFO][1][MainProcess][MainThread] - core.schains.cleaner:65 - Cleaner process started [2023-09-07 17:13:07,761 INFO][30636][cleaner][MainThread] - core.schains.cleaner:110 - Cleaner procedure started. [2023-09-07 17:13:07,933 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:13:07,933 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:13:07,935 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:13:07.935541 [2023-09-07 17:13:07,943 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:13:07,943 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:13:07,949 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:13:07,952 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694106787.942308, 'te': 1694106787.9527495, 'initial_status': True} ====================================================================================================  [2023-09-07 17:13:07,954 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:13:07.954484 [2023-09-07 17:13:07,959 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:13:07,960 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:13:12,020 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106780, schain_finish_ts: 1692637668 [2023-09-07 17:13:14,003 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:13:14,004 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:13:14,004 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:13:14,004 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:13:14,005 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:13:14,005 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:13:14,009 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:13:14,011 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:13:14,036 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:13:14,280 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:13:14,493 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:13:14,577 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 17:13:14,578 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 17:13:14,597 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:13:14,598 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2501 [2023-09-07 17:13:14,599 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:13:14,599 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:13:14,599 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:13:14,600 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:13:14,600 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:13:14,607 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:13:14,609 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:13:14.609317 [2023-09-07 17:13:14,790 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:13:14,818 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:13:14,846 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:13:14,848 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:13:14,854 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:13:14,855 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694106794.816, 'te': 1694106794.8224213, 'initial_status': True} ====================================================================================================  [2023-09-07 17:13:14,856 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:13:14.856822 [2023-09-07 17:13:14,861 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:13:14,862 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:13:17,221 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:13:17,221 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:13:17,223 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:13:17.223553 [2023-09-07 17:13:17,230 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:13:17,230 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:13:17,236 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:13:17,240 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694106797.2292254, 'te': 1694106797.240334, 'initial_status': True} ====================================================================================================  [2023-09-07 17:13:17,242 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:13:17.242137 [2023-09-07 17:13:17,246 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:13:17,246 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:13:20,390 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106792, schain_finish_ts: 1692377364 [2023-09-07 17:13:23,368 INFO][30636][cleaner][MainThread] - core.schains.cleaner:116 - sChains on contracts: ['portly-passionate-sirius', 'curly-red-alterf', 'howling-hoedus', 'green-giddy-denebola', 'wan-tasty-kitalpha', 'free-basic-rosy-SKALE', 'submissive-well-off-rotanev'] sChains on node: ['curly-red-alterf', 'free-basic-rosy-SKALE', 'green-giddy-denebola', 'howling-hoedus', 'portly-passionate-sirius', 'submissive-well-off-rotanev', 'wan-tasty-kitalpha'] [2023-09-07 17:13:23,369 INFO][30636][cleaner][MainThread] - core.schains.cleaner:132 - Cleanup procedure finished [2023-09-07 17:13:23,380 INFO][1][MainProcess][MainThread] - core.schains.cleaner:67 - Cleaner process is joined. [2023-09-07 17:13:23,380 INFO][1][MainProcess][MainThread] - core.schains.cleaner:68 - Terminating the process [2023-09-07 17:13:23,381 INFO][1][MainProcess][MainThread] - __main__:70 - Sleeping for 90s after run_cleaner [2023-09-07 17:13:26,508 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:13:26,508 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:13:26,509 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:13:26,509 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:13:26,509 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:13:26,509 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:13:26,510 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:13:26,510 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 17:13:26,548 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:13:26,749 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:13:26,925 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:13:27,030 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:13:27,031 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2486 [2023-09-07 17:13:27,032 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:13:27,032 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:13:27,032 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:13:27,033 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:13:27,033 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:13:27,039 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:13:27,041 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:13:27.041431 [2023-09-07 17:13:27,182 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.main:162 - Starting dkg procedure for free-basic-rosy-SKALE [2023-09-07 17:13:27,230 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:13:27,260 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:13:27,294 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:13:27,297 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:13:27,304 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:13:27,304 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694106807.2580492, 'te': 1694106807.2650619, 'initial_status': True} ====================================================================================================  [2023-09-07 17:13:27,306 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:13:27.306694 [2023-09-07 17:13:27,314 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:13:27,320 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:13:29,660 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:77 - sChain free-basic-rosy-SKALE: Sending broadcast [2023-09-07 17:13:29,987 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:13:29,987 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:13:29,989 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:13:29.989634 [2023-09-07 17:13:29,995 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:13:29,996 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:13:30,001 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:13:30,005 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694106809.994588, 'te': 1694106810.0051887, 'initial_status': True} ====================================================================================================  [2023-09-07 17:13:30,007 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:13:30.007209 [2023-09-07 17:13:30,011 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:13:30,012 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:13:30,044 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'isPolyExists', 'params': {'polyName': 'POLY:SCHAIN_ID:18860195493846376099765944407454631582519828060563062387235736098366005:NODE_ID:14:DKG_ID:2'}} [2023-09-07 17:13:30,210 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'IsExist': False, 'errorMessage': '', 'status': 0}} [2023-09-07 17:13:30,210 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'generateDKGPoly', 'params': {'polyName': 'POLY:SCHAIN_ID:18860195493846376099765944407454631582519828060563062387235736098366005:NODE_ID:14:DKG_ID:2', 't': 11}} [2023-09-07 17:13:30,388 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'status': 0}} [2023-09-07 17:13:38,282 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106792, schain_finish_ts: 1693742304 [2023-09-07 17:13:42,418 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getVerificationVector', 'params': {'polyName': 'POLY:SCHAIN_ID:18860195493846376099765944407454631582519828060563062387235736098366005:NODE_ID:14:DKG_ID:2', 'n': 16, 't': 11}} [2023-09-07 17:13:42,646 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'status': 0, 'verificationVector': [['9236134442235288382953177031521034499481448580548187907897830021776262929712', '2559865065054789879984292968128578838806487660460494172008051012902991856234', '9558945528789288043691181928271922232624660353475109819562559543049198017226', '20865966848585219611176867539849396932288280464796652802724232858239754101191'], ['18041309823557232113808866714328383160752215869136385827571137143477575585730', '12384844897855532620324017444100952049814049256804027649581803783210342134015', '18305340914313954071944588475111257035246909321653935250157260650207030379715', '17619663613011385419040671796169117004476017264038160853779304582268617603531'], ['4190886929731728407283838749364001400214087266158052894278356414742850912730', '18486657704035621263738819983779927779087599134727705373030636086974477370671', '5162080827228086872950262086701927857299813411127763158265760299156110583056', '6115110793734624122972085154776552055440287070025606670664673186406505204250'], ['508118252629176855119580305982349004632578869406522455511931669721649675414', '9676166235097914578938531039269176362298117512595992622253845783395043022811', '16420522286712067051337685159775454080144346043346107964769797392375476482890', '13642273036411260638139046030080304162226437891715451324013858459530999221404'], ['3439407870421402584614262225388496309596972410262177453065633234894523036514', '14180250255521358548265857860526525146785479146621098245003198889045409287101', '15508830602246844023294987686878901708822032552373698460440986389957295229654', '20524141293818871682109716121726777457962003750383268026426740035083232426684'], ['1327878350519644428083761361368803359631404769437233434945222604243853928442', '20434631665819872281393829878112183939349642268176182476404512921929177706664', '852634407599919005861730277393516355495288510222938004058133998131204892260', '35338791012330255676594720560733400263146935484931797677642554899404924309'], ['17955339701204048512816634920701005147743418861319273907432779771029765119113', '18236001703191456156784308650859042452761510450369384477977995326543452285350', '12067925826138572328445612888340358160033396462385754404262722778264995433840', '18068806179774797588900303639401403080958435009930145275329984647821233570359'], ['4606730248743534125298884252405414401141365094494448497772034264463752215790', '4027242898867932438906609086819773043737986249590039304982028753984162838222', '20656034428619129320241790349647642258376765611420664227464909173239352145907', '19627733258077891671568661938983413951067470670622198943432762674970839502223'], ['4480312420431055954169038594272012209860162765160784760600308224241005531865', '11551227849829352003901680280925435941058084777452861707354286196912980859609', '11170297319356066034974553651115739190726360139449092913213308236489710772530', '11713591813156434624455449778504276877623227372079345484830198842313844516651'], ['5669941103404423021235756134964595311632661806132797566426961066979190286042', '6820046714983184052549032395403239956201804135872078077222188233082677491672', '1146048909473778741625912493995169048673302133142178271876314788609451365523', '19345093489842706685861215010370963644432514350472251846652029644674475330335'], ['4868930441781704845961052940730612849216255175559399367679413862794856504126', '8173867675764662779679682961894048608354474699300368036953830015123511061684', '2313787822572353817648076926513506475420680230151580222312959490136853499485', '19976809884674859815517564008102284401057509145565902686875718986129212666537']]}} [2023-09-07 17:13:42,647 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getSecretShareV2', 'params': {'polyName': 'POLY:SCHAIN_ID:18860195493846376099765944407454631582519828060563062387235736098366005:NODE_ID:14:DKG_ID:2', 'n': 16, 't': 11, 'publicKeys': ['0dac4abf55c382485f63b112f1d381269c5c30ac8ac797b1da9f025a960506b767442c5b443dfc8cc69a7d12bbf4b9c745c8cb7a41f2a6c07c6ad9923d74c1a5', '6a361515e86bf759bf3e6ab5ff876a428570410f57788ee8bc135e7111f94e9feaee770d62f322aee9a3b19acb416b8b6c5228df3185b3a2b2e4d5688aed5a73', '60edf27e91cae9219f16532e9959544e3b3ae4fece0afbad34ea83ddc44581736efc51b60cadb8ebb20a9764dc9c6471744f55caa9642f6310e46b97ff8491bb', '9e2be42ee5f198709fb3befd78057fff039f109bb09c37152180bef985e65bb23871e7025983d1500dfdd4a7ea94e87e6ce4a0c73a57c25e6dabd024b08a09a4', 'ce2c88352efc9ceddddcca42bd528e76f772a528ae2ea12a34d72062b2d6ce86f6da04711f9d8de530c50f3f053baca939609419f263ffee324f6a11f450cc00', 'c9d428e666721c70ec30328ea62efb245e26aaff4b560428c4b8f2c7f3d598b870ba1161a1d1f22883fe6c52d7cac874c68d292c294f92bc990838e4734316fe', 'ce60731bf92f17420db5703b2d9c1f410f3f7a14096c20681e3bd3f372cc5ce98b34120b819091993362efa7f121f611a82697019c120c7e43cba90016f3d3c0', 'c3771ddbfd86eaea3eb95cb491cd57de7b0246da912fd3cb00d26c545baf35298b7d53ca817f5033a8e756e8b4baaa0fe00813e48ae1ffece525530b5a36fefa', 'e05064f22cf9c4ae3103e362aadbbc788b8044d8e8198bd9c3aceb836c716f98ba8a3ae9881e33abd81bfba22e77dde561feaa283c8e77960dc5b8f5a620cd81', '1dc327e01eca315119db6eeddbc9216a9069584e2e5185485f3b6aea894dc79a65db549a6e2b4943488df48c527a1860eaaea7b07b16e25447a1a1e247a58628', 'c549e448f3e779a6f48e1e555a891ef721934c67b6d6be3aa387d4ba707c3d237688788cce3ff77a5731a85d2d35fdfcd99a721a59f1ad8ca28efcca61663705', '53265d123f2f1b6462df6b92141256a7dbfa002b87241147b16591ae037756ac4e77ba24499a097995b86c0c87e87d902211953cd6a616b89b04c38714970d8a', 'd40aa8bfbaa0d621910012f05afeda8302662bec1a862f67c092e972a968be66da826501acab258dab7fc2ad6d9873d3596adefe1bb38f47e11b6e73a36482b7', '49351716ba3fe96d9fa4c1357a6039fffbff32ceb97cb05ee1e318275299a413bdfc780b8cd12d5a36ba0260d41853de742be925eb0df043a5a8ab7f54de34ec', '656dcde97a1244e245e36dae1c6c1d4e418fedf94a09059057118e1a3a96d5b676ddd2f44cfc4faf5f820363f2d9843443c3cc262d0d48b2d633837e5eb369d9', '22753df837fa486d41434916b76ca1c61d91f367fdda3654eb813b6fcac423fb8c4aa487203e1771c690c337073d52b52744d4d2cf5d7c0d579145c94e943a43']}} [2023-09-07 17:13:42,927 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'secretShare': '4039eb8f357618bfdf4ddb4bede20e1d0fa2a9e7a525cdb229...', 'status': 0}} [2023-09-07 17:13:48,701 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106792, schain_finish_ts: 1693742004 [2023-09-07 17:13:48,747 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.transactions.tools:46 - Dry run tx: broadcast, sender: 0x523E13D962A7614752eFa3dB62e0466dF0d9E89D, wallet: RedisWalletAdapter, value: 0, [2023-09-07 17:13:53,545 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106828, schain_finish_ts: 1692637668 [2023-09-07 17:13:55,426 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:13:55,426 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:13:55,426 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:13:55,427 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 17:13:55,427 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 17:13:55,427 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:13:55,428 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:13:55,429 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:13:55,461 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:13:55,683 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:13:55,886 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:13:55,985 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:13:55,986 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2508 [2023-09-07 17:13:55,986 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:13:55,986 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:13:55,987 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:13:55,987 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:13:55,987 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:13:55,994 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:13:55,996 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:13:55.996277 [2023-09-07 17:13:56,233 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:13:56,261 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:13:56,294 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:13:56,296 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:13:56,305 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:13:56,306 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694106836.2598102, 'te': 1694106836.2665834, 'initial_status': True} ====================================================================================================  [2023-09-07 17:13:56,307 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:13:56.307953 [2023-09-07 17:13:56,312 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:13:56,313 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:13:56,736 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106828, schain_finish_ts: 0 [2023-09-07 17:13:56,745 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106828, schain_finish_ts: 1694108304 [2023-09-07 17:14:01,823 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106840, schain_finish_ts: 1692377364 [2023-09-07 17:14:01,874 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106840, schain_finish_ts: 1693412520 [2023-09-07 17:14:02,167 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:14:02,168 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:14:02,168 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:14:02,168 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:14:02,169 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:14:02,169 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:14:02,182 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.transactions.tools:60 - Estimated gas for broadcast: 624562 [2023-09-07 17:14:02,203 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:14:02,389 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.transactions.tools:126 - Tx: broadcast, Fields: {'chainId': 5, 'gas': 624562, 'nonce': 312, 'value': 0, 'gasPrice': 10000000000, 'type': 1, 'to': '0x69E2C9576B36c8d7938Ba45a1C909B67EC53048E', 'data': '0x7d93019b000002bb9043962a56bacebbde03191f32a67a4b888426ac09f83548ae92e235000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000620000000000000000000000000000000000000000000000000000000000000000b146b7782d130c96bf20522f40e413fbb479ee7420701b050c5a898afeb419d3005a8d510713b4848a2d6525eda8681376e8a7e69cd1e402303b7ddd6666a826a15222be37ea57af2b6b281a3b32ae5bc79be4120049703ff49ea86d744649aca2e21b7fdbf84e02dd2a774e6cc883d8e5d4cf321e2432d7c2874036a1ba0e1c727e304fda3d4576d46f6959e03a5ec3812a90eb18315485a45a4ea7c54138fc21b6192e4b450970d23f9cbc864a8cb600b621ae07ea73f5b54ce33d11d71b0ff287874af1918c5ab3bae0ebc800658624454681f32aa10122b77dcc3984184c326f4604e76d93576cce27997959d8a0ee13ec94d8344680abb7b837dde41b9cb0943f53bc4163fe43b0ad0e9c5acf43182ea8f1d6c651d2b772bec7199af41da28df13d49d57b0c3de13fb2ea9999e738e805fc7fb364e293a00466965d2052f0b69a25b8c083686e857628d4248ed32aeed38303fb3e55f086012a2434425100d8507b1aebbeddffc10ee01f09b307a58820c10499ca8f33b53a9086b7d4a1a011f95b17ab28b7b0106c05fe393e85b8eaf1198aaf64605e8b783583fe67c9615648419249b14a1f1454558dc08ad57b6fb087adbb03878da2589e4b10a97db244dafaea23ba8653ed3b59dec69b4979ee5f66196988b0780964b68c7c5734a1e2940be8f65363ff2083cdadc5d4e79edb1953155835c9d69c6a22fc4d5cc9c079aa2c846e2c83a4a07f8c5feebf3981cabb8a2890d64713c97320cc5f173621f59bcbaa92179469d99539473192cd977b24cd3be869626f2c2021664a24bbd2249afe29925cc2c9c7c4e1ba96c484e1af83093f6b8e524781ac72563f84ad62d6040973f00d5d26f39a496b7e9281d8c582be6b2630659196730f30e0d3abc02ef8d71737caff59cacf6ad393b78d5d6e835402d3daa28e2d62ea384cc67fa2d2d97769e70e5c0e480fc4be365ed83509069fd380eb63d22950aa3fb7530a801e292efe3bb15fe201563e508c21dbb3911e74541e8d42e6f2a553f2430a664001400449c1c5f517dd956b67b28d98f93f1781ff5a956cd3173f571764ba59527b25cb492e9de1b727712c16d047b77867f785ac10de2730dc9584005ea5c8928513610d0720f4bb7aaab0f664254d6822cb2047d154a17557cd205cec1a9a61aae343700e731939f4dec4e18ea89c38904d42adb2c95ac8c2e93e5e7450d7027f294f62d0e0da43972a30f5fc2f4a1ad72a7fa72233e8b9116755f83e142370a2f5121bb62970bf78d68718860450636adfc6fbafb42740fac3f3a148ebcee08e756b54ed49da3c83197f138e46e4a1763e60d6a6758542e09f233927f36ce2daae6b383b3220d70949d8889b9b43f21361a1526fd7b3a57050811480037f32b64e7413fc80453492b8a34255f270536b81a0cc55a8043de2909c79a184d8f09e7c457de5f453390f227cf9457e074c3722a2347d1c3a337621830f9ac92d91989c365e2c30c57e5cdda373173b1fd208a2c69ec3558d15245809b1f319ad918b22a0b38397f9069ef8bae5d48cfa4114796f7ad5051645514f0770eda213219e5a8740001187890e38cbc49909f4073056dfd3e8113a4807ec05618bce72b0c8912ac25d84264a6df22a8b61e3e935592bfe996be9c14c36a104f7b26a6da0f1402783b5e909532c345229435cf58c918bb806ee2e4bdbd8eb8a9f8cff7d80288a404a3dcc690c0e09326cce4395028fa6725118168f0263ee25be54014932ac4ef5594dbb77c90d40d5083c98dc918787d5f4dc99045ae5845c0c6058b1f0ac3b78b6b5492be28f100496a1ecfa0d81768c5385ab38489942ca451c81b3e12123ec294197bbff94cb9d6eff6fbeefb1dbbd4fe6efbe79e245130f4d82cb4051d8eb83a8cc23e7a88f6511be0bbd9fd44a646d4ddbbf37f6566d05144ce5d2c2a7944f418187f6a7345f3ec24ff257a0f244b74ff218d7b77ebac7fb372a9000000000000000000000000000000000000000000000000000000000000001095a4aa7fc33f9b7b18122602d01cc2f955018df3fea1ceca5511e95b42bc9db7ed5295549365e14f33f67d9f56c62a8500e1e33db292f7ac381c5bbcbe9357404039eb8f357618bfdf4ddb4bede20e1d0fa2a9e7a525cdb229638d5a74b58fafa25e8e953c5371917c366ad84534583bd044f79b989b1365c599da10b87b402fd58a20003417738bc16a2acb8369b19eba919850e0510a8ca95bd59ca575902a1aee3e0e0716559545f5a86bd9e9de6291b71df66fef02222d1069b3d8b9edb9ea492276c13f9363f765f8dcb27615f4ebd71c580f105ad69777cd8e1f099dd073da31cd615937ece506d60f2b8ccb321557dd11d647e00e2ccf548f6e88b0071e45932351f0e762f65648a4b084ababc28a2ab319b05c549459b5e858354c9f026b775ba8cc93bb6afb28acf01edec65f637a342311c8962d006eec1ee2c1dffe75b27b9272ab21b6a34b17b3410d65299fcc5b24f0b4a53eafc4fba443b4ca084f44061d44e80961a0d8e4627a79142412d4f61bb763d5a1dc9b74e8b1c0ecf64c0fa0f60915ba4af25251563818313ecad9b663c0706158cf26cacfa8f178abc0564b8d20bfa1e41d5afe22f041a668c039da0caab718668489cf6da81986e91c0e02cb7d9d01ced06c3fa33a15b747aa733c9f04e336bf1450bb4578c679c02e0c352266cb66c12cbd69615b825f22e5df70a1119b7a432bb60837fba164bf762e63b49259a874e0017e863db70fbcffa6c7a934f0b51ae82bfa1908dc0dfbefcd22ac959ce6c845103c53b523666d890ac6d00a11e10490f3a5a04339a92c27252d33af77c711dbb4611668cb1dce29d967eacc5011ceb72334c4b26d9a96d9bec48ad84d8648a6fb9c37f883a1c2cab9118b5f00391cc333e28826edde8375953cf2f1a398bfd36a37e938509e914c19efa04fc35a3075ea149cc469cad140e73d6ec9ab8f2d1f1593a46d1963c3000a4c96be0b5363bbbfca5b5fbc70b74f9e26e525d14a2fdcce142fb8a227994a52e4cc3bc21d037f99b7a0afd7ea42c5c6af4c72f3945c6863eb86fdffdf7bb7a9df794508b8cc7a463c9f4df1b2157a8dcfb69bc67705ab3a351eae8b1387a61f78cb667236f12ca725f862044386bff2b781fbe715d52f6066f1f26ebf17ea1186de7f04b6f16f68c88a404c302e7efdf5de39b46c05cf86f37f6d9e1f1b2a19c4415dea7b881c1e69c6d54d3e1c7e7d466a8a78a7a3b5e4d5ad52caa47ce0901fa3bbf593b0ee523ef5854fd447ca77a6d2bf8b49869210b8d363628d4b6cdd853ff0276c9a95d58adca9a4280bb5c0bd55f4a1e8ec5102dd8eb132fc0b4b44178dc2f72ad015276e792f0cd0102d26ef550dfb7902595992ad15a676e6ff2c038d9d49b3affb197cb39a7197ca52867cc1237f924a2272e9b8b0ea8149d216d58abf72d023f89b2c5ebc3bee79dbdfe4ba64454c9b2e2095dbf613aacbbcbadb18118c245fdccfda8770139e26a1e44002d8d0dcd69ef4c715dcbc4471ce1c5d5298e2343ffbcd0a638aaad8fe87884642b9d1d435e9fec696a788b9848ce9165a3be37e8a7d9bc4549c2f3c9bb12c0b71e681046ebaa1096bd2faa9f682a13dfa178046e0346da4a5adf68734198abcc4d8b688746ec64d64f9bbc3ee37b6266dee17c6bd6c39ae39023ad56c2d89a4e7aba46abf9c6ce542ccae6da71727c851d9517b48fcbe52f3c02aca30d75d54df626181d224d8f909aab233f0beefdcdf4e8de99829f99086460a6c6f243bcd17b833ef22f4d75b5d9d779f6e7c15efb707150f781df054b5d64d217b16e0a27976fa32768d0b0f22b56d9a2a581346bc858209f92b506363821d9bba6823048b4da4304756176e3a008e7859e3179c4e159140471656e146fcbfef55eba1381e6a09bb9a81bf26a9eaff56ebfcb4406c5194c6c74ae976741121c797f5bad475b0d35e355764125e7b67b6d3c74d7bc4ff4195dd3c8a3b3e7c41334e7053fe9a365b24193562ffa35da82bfaaae2a2332e2bc02e497bee0a710be13f9ecfa30a6391f64bf29b0b86e1a4cacb7d71dc7f92210dc36ef5b951a426ab81ad5fc202d3b636750630339069ba7b8450ffc87abf4fa6715ab88f0918d807b3dedfdc7ef405792c4e9905b435a1cc66d76abffecfcaa6cd2aacc590961d40'}, [2023-09-07 17:14:02,390 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.wallets.redis_wallet:135 - Sending {'chainId': 5, 'gas': 624562, 'nonce': 312, 'value': 0, 'gasPrice': 10000000000, 'type': 1, 'to': '0x69E2C9576B36c8d7938Ba45a1C909B67EC53048E', 'data': '0x7d93019b000002bb9043962a56bacebbde03191f32a67a4b888426ac09f83548ae92e235000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000620000000000000000000000000000000000000000000000000000000000000000b146b7782d130c96bf20522f40e413fbb479ee7420701b050c5a898afeb419d3005a8d510713b4848a2d6525eda8681376e8a7e69cd1e402303b7ddd6666a826a15222be37ea57af2b6b281a3b32ae5bc79be4120049703ff49ea86d744649aca2e21b7fdbf84e02dd2a774e6cc883d8e5d4cf321e2432d7c2874036a1ba0e1c727e304fda3d4576d46f6959e03a5ec3812a90eb18315485a45a4ea7c54138fc21b6192e4b450970d23f9cbc864a8cb600b621ae07ea73f5b54ce33d11d71b0ff287874af1918c5ab3bae0ebc800658624454681f32aa10122b77dcc3984184c326f4604e76d93576cce27997959d8a0ee13ec94d8344680abb7b837dde41b9cb0943f53bc4163fe43b0ad0e9c5acf43182ea8f1d6c651d2b772bec7199af41da28df13d49d57b0c3de13fb2ea9999e738e805fc7fb364e293a00466965d2052f0b69a25b8c083686e857628d4248ed32aeed38303fb3e55f086012a2434425100d8507b1aebbeddffc10ee01f09b307a58820c10499ca8f33b53a9086b7d4a1a011f95b17ab28b7b0106c05fe393e85b8eaf1198aaf64605e8b783583fe67c9615648419249b14a1f1454558dc08ad57b6fb087adbb03878da2589e4b10a97db244dafaea23ba8653ed3b59dec69b4979ee5f66196988b0780964b68c7c5734a1e2940be8f65363ff2083cdadc5d4e79edb1953155835c9d69c6a22fc4d5cc9c079aa2c846e2c83a4a07f8c5feebf3981cabb8a2890d64713c97320cc5f173621f59bcbaa92179469d99539473192cd977b24cd3be869626f2c2021664a24bbd2249afe29925cc2c9c7c4e1ba96c484e1af83093f6b8e524781ac72563f84ad62d6040973f00d5d26f39a496b7e9281d8c582be6b2630659196730f30e0d3abc02ef8d71737caff59cacf6ad393b78d5d6e835402d3daa28e2d62ea384cc67fa2d2d97769e70e5c0e480fc4be365ed83509069fd380eb63d22950aa3fb7530a801e292efe3bb15fe201563e508c21dbb3911e74541e8d42e6f2a553f2430a664001400449c1c5f517dd956b67b28d98f93f1781ff5a956cd3173f571764ba59527b25cb492e9de1b727712c16d047b77867f785ac10de2730dc9584005ea5c8928513610d0720f4bb7aaab0f664254d6822cb2047d154a17557cd205cec1a9a61aae343700e731939f4dec4e18ea89c38904d42adb2c95ac8c2e93e5e7450d7027f294f62d0e0da43972a30f5fc2f4a1ad72a7fa72233e8b9116755f83e142370a2f5121bb62970bf78d68718860450636adfc6fbafb42740fac3f3a148ebcee08e756b54ed49da3c83197f138e46e4a1763e60d6a6758542e09f233927f36ce2daae6b383b3220d70949d8889b9b43f21361a1526fd7b3a57050811480037f32b64e7413fc80453492b8a34255f270536b81a0cc55a8043de2909c79a184d8f09e7c457de5f453390f227cf9457e074c3722a2347d1c3a337621830f9ac92d91989c365e2c30c57e5cdda373173b1fd208a2c69ec3558d15245809b1f319ad918b22a0b38397f9069ef8bae5d48cfa4114796f7ad5051645514f0770eda213219e5a8740001187890e38cbc49909f4073056dfd3e8113a4807ec05618bce72b0c8912ac25d84264a6df22a8b61e3e935592bfe996be9c14c36a104f7b26a6da0f1402783b5e909532c345229435cf58c918bb806ee2e4bdbd8eb8a9f8cff7d80288a404a3dcc690c0e09326cce4395028fa6725118168f0263ee25be54014932ac4ef5594dbb77c90d40d5083c98dc918787d5f4dc99045ae5845c0c6058b1f0ac3b78b6b5492be28f100496a1ecfa0d81768c5385ab38489942ca451c81b3e12123ec294197bbff94cb9d6eff6fbeefb1dbbd4fe6efbe79e245130f4d82cb4051d8eb83a8cc23e7a88f6511be0bbd9fd44a646d4ddbbf37f6566d05144ce5d2c2a7944f418187f6a7345f3ec24ff257a0f244b74ff218d7b77ebac7fb372a9000000000000000000000000000000000000000000000000000000000000001095a4aa7fc33f9b7b18122602d01cc2f955018df3fea1ceca5511e95b42bc9db7ed5295549365e14f33f67d9f56c62a8500e1e33db292f7ac381c5bbcbe9357404039eb8f357618bfdf4ddb4bede20e1d0fa2a9e7a525cdb229638d5a74b58fafa25e8e953c5371917c366ad84534583bd044f79b989b1365c599da10b87b402fd58a20003417738bc16a2acb8369b19eba919850e0510a8ca95bd59ca575902a1aee3e0e0716559545f5a86bd9e9de6291b71df66fef02222d1069b3d8b9edb9ea492276c13f9363f765f8dcb27615f4ebd71c580f105ad69777cd8e1f099dd073da31cd615937ece506d60f2b8ccb321557dd11d647e00e2ccf548f6e88b0071e45932351f0e762f65648a4b084ababc28a2ab319b05c549459b5e858354c9f026b775ba8cc93bb6afb28acf01edec65f637a342311c8962d006eec1ee2c1dffe75b27b9272ab21b6a34b17b3410d65299fcc5b24f0b4a53eafc4fba443b4ca084f44061d44e80961a0d8e4627a79142412d4f61bb763d5a1dc9b74e8b1c0ecf64c0fa0f60915ba4af25251563818313ecad9b663c0706158cf26cacfa8f178abc0564b8d20bfa1e41d5afe22f041a668c039da0caab718668489cf6da81986e91c0e02cb7d9d01ced06c3fa33a15b747aa733c9f04e336bf1450bb4578c679c02e0c352266cb66c12cbd69615b825f22e5df70a1119b7a432bb60837fba164bf762e63b49259a874e0017e863db70fbcffa6c7a934f0b51ae82bfa1908dc0dfbefcd22ac959ce6c845103c53b523666d890ac6d00a11e10490f3a5a04339a92c27252d33af77c711dbb4611668cb1dce29d967eacc5011ceb72334c4b26d9a96d9bec48ad84d8648a6fb9c37f883a1c2cab9118b5f00391cc333e28826edde8375953cf2f1a398bfd36a37e938509e914c19efa04fc35a3075ea149cc469cad140e73d6ec9ab8f2d1f1593a46d1963c3000a4c96be0b5363bbbfca5b5fbc70b74f9e26e525d14a2fdcce142fb8a227994a52e4cc3bc21d037f99b7a0afd7ea42c5c6af4c72f3945c6863eb86fdffdf7bb7a9df794508b8cc7a463c9f4df1b2157a8dcfb69bc67705ab3a351eae8b1387a61f78cb667236f12ca725f862044386bff2b781fbe715d52f6066f1f26ebf17ea1186de7f04b6f16f68c88a404c302e7efdf5de39b46c05cf86f37f6d9e1f1b2a19c4415dea7b881c1e69c6d54d3e1c7e7d466a8a78a7a3b5e4d5ad52caa47ce0901fa3bbf593b0ee523ef5854fd447ca77a6d2bf8b49869210b8d363628d4b6cdd853ff0276c9a95d58adca9a4280bb5c0bd55f4a1e8ec5102dd8eb132fc0b4b44178dc2f72ad015276e792f0cd0102d26ef550dfb7902595992ad15a676e6ff2c038d9d49b3affb197cb39a7197ca52867cc1237f924a2272e9b8b0ea8149d216d58abf72d023f89b2c5ebc3bee79dbdfe4ba64454c9b2e2095dbf613aacbbcbadb18118c245fdccfda8770139e26a1e44002d8d0dcd69ef4c715dcbc4471ce1c5d5298e2343ffbcd0a638aaad8fe87884642b9d1d435e9fec696a788b9848ce9165a3be37e8a7d9bc4549c2f3c9bb12c0b71e681046ebaa1096bd2faa9f682a13dfa178046e0346da4a5adf68734198abcc4d8b688746ec64d64f9bbc3ee37b6266dee17c6bd6c39ae39023ad56c2d89a4e7aba46abf9c6ce542ccae6da71727c851d9517b48fcbe52f3c02aca30d75d54df626181d224d8f909aab233f0beefdcdf4e8de99829f99086460a6c6f243bcd17b833ef22f4d75b5d9d779f6e7c15efb707150f781df054b5d64d217b16e0a27976fa32768d0b0f22b56d9a2a581346bc858209f92b506363821d9bba6823048b4da4304756176e3a008e7859e3179c4e159140471656e146fcbfef55eba1381e6a09bb9a81bf26a9eaff56ebfcb4406c5194c6c74ae976741121c797f5bad475b0d35e355764125e7b67b6d3c74d7bc4ff4195dd3c8a3b3e7c41334e7053fe9a365b24193562ffa35da82bfaaae2a2332e2bc02e497bee0a710be13f9ecfa30a6391f64bf29b0b86e1a4cacb7d71dc7f92210dc36ef5b951a426ab81ad5fc202d3b636750630339069ba7b8450ffc87abf4fa6715ab88f0918d807b3dedfdc7ef405792c4e9905b435a1cc66d76abffecfcaa6cd2aacc590961d40'} to redis pool, method: dkg.broadcast [2023-09-07 17:14:02,395 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.wallets.redis_wallet:145 - Adding tx b'tx-28acc9e026d8e06c' to the pool [2023-09-07 17:14:02,397 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.wallets.redis_wallet:147 - Saving tx b'tx-28acc9e026d8e06c' record: b'{"status": "PROPOSED", "score": 51694106842, "multiplier": null, "tx_hash": null, "method": "dkg.broadcast", "meta": null, "chainId": 5, "gas": null, "nonce": 312, "value": 0, "gasPrice": 10000000000, "type": 1, "to": "0x69E2C9576B36c8d7938Ba45a1C909B67EC53048E", "data": "0x7d93019b000002bb9043962a56bacebbde03191f32a67a4b888426ac09f83548ae92e235000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000620000000000000000000000000000000000000000000000000000000000000000b146b7782d130c96bf20522f40e413fbb479ee7420701b050c5a898afeb419d3005a8d510713b4848a2d6525eda8681376e8a7e69cd1e402303b7ddd6666a826a15222be37ea57af2b6b281a3b32ae5bc79be4120049703ff49ea86d744649aca2e21b7fdbf84e02dd2a774e6cc883d8e5d4cf321e2432d7c2874036a1ba0e1c727e304fda3d4576d46f6959e03a5ec3812a90eb18315485a45a4ea7c54138fc21b6192e4b450970d23f9cbc864a8cb600b621ae07ea73f5b54ce33d11d71b0ff287874af1918c5ab3bae0ebc800658624454681f32aa10122b77dcc3984184c326f4604e76d93576cce27997959d8a0ee13ec94d8344680abb7b837dde41b9cb0943f53bc4163fe43b0ad0e9c5acf43182ea8f1d6c651d2b772bec7199af41da28df13d49d57b0c3de13fb2ea9999e738e805fc7fb364e293a00466965d2052f0b69a25b8c083686e857628d4248ed32aeed38303fb3e55f086012a2434425100d8507b1aebbeddffc10ee01f09b307a58820c10499ca8f33b53a9086b7d4a1a011f95b17ab28b7b0106c05fe393e85b8eaf1198aaf64605e8b783583fe67c9615648419249b14a1f1454558dc08ad57b6fb087adbb03878da2589e4b10a97db244dafaea23ba8653ed3b59dec69b4979ee5f66196988b0780964b68c7c5734a1e2940be8f65363ff2083cdadc5d4e79edb1953155835c9d69c6a22fc4d5cc9c079aa2c846e2c83a4a07f8c5feebf3981cabb8a2890d64713c97320cc5f173621f59bcbaa92179469d99539473192cd977b24cd3be869626f2c2021664a24bbd2249afe29925cc2c9c7c4e1ba96c484e1af83093f6b8e524781ac72563f84ad62d6040973f00d5d26f39a496b7e9281d8c582be6b2630659196730f30e0d3abc02ef8d71737caff59cacf6ad393b78d5d6e835402d3daa28e2d62ea384cc67fa2d2d97769e70e5c0e480fc4be365ed83509069fd380eb63d22950aa3fb7530a801e292efe3bb15fe201563e508c21dbb3911e74541e8d42e6f2a553f2430a664001400449c1c5f517dd956b67b28d98f93f1781ff5a956cd3173f571764ba59527b25cb492e9de1b727712c16d047b77867f785ac10de2730dc9584005ea5c8928513610d0720f4bb7aaab0f664254d6822cb2047d154a17557cd205cec1a9a61aae343700e731939f4dec4e18ea89c38904d42adb2c95ac8c2e93e5e7450d7027f294f62d0e0da43972a30f5fc2f4a1ad72a7fa72233e8b9116755f83e142370a2f5121bb62970bf78d68718860450636adfc6fbafb42740fac3f3a148ebcee08e756b54ed49da3c83197f138e46e4a1763e60d6a6758542e09f233927f36ce2daae6b383b3220d70949d8889b9b43f21361a1526fd7b3a57050811480037f32b64e7413fc80453492b8a34255f270536b81a0cc55a8043de2909c79a184d8f09e7c457de5f453390f227cf9457e074c3722a2347d1c3a337621830f9ac92d91989c365e2c30c57e5cdda373173b1fd208a2c69ec3558d15245809b1f319ad918b22a0b38397f9069ef8bae5d48cfa4114796f7ad5051645514f0770eda213219e5a8740001187890e38cbc49909f4073056dfd3e8113a4807ec05618bce72b0c8912ac25d84264a6df22a8b61e3e935592bfe996be9c14c36a104f7b26a6da0f1402783b5e909532c345229435cf58c918bb806ee2e4bdbd8eb8a9f8cff7d80288a404a3dcc690c0e09326cce4395028fa6725118168f0263ee25be54014932ac4ef5594dbb77c90d40d5083c98dc918787d5f4dc99045ae5845c0c6058b1f0ac3b78b6b5492be28f100496a1ecfa0d81768c5385ab38489942ca451c81b3e12123ec294197bbff94cb9d6eff6fbeefb1dbbd4fe6efbe79e245130f4d82cb4051d8eb83a8cc23e7a88f6511be0bbd9fd44a646d4ddbbf37f6566d05144ce5d2c2a7944f418187f6a7345f3ec24ff257a0f244b74ff218d7b77ebac7fb372a9000000000000000000000000000000000000000000000000000000000000001095a4aa7fc33f9b7b18122602d01cc2f955018df3fea1ceca5511e95b42bc9db7ed5295549365e14f33f67d9f56c62a8500e1e33db292f7ac381c5bbcbe9357404039eb8f357618bfdf4ddb4bede20e1d0fa2a9e7a525cdb229638d5a74b58fafa25e8e953c5371917c366ad84534583bd044f79b989b1365c599da10b87b402fd58a20003417738bc16a2acb8369b19eba919850e0510a8ca95bd59ca575902a1aee3e0e0716559545f5a86bd9e9de6291b71df66fef02222d1069b3d8b9edb9ea492276c13f9363f765f8dcb27615f4ebd71c580f105ad69777cd8e1f099dd073da31cd615937ece506d60f2b8ccb321557dd11d647e00e2ccf548f6e88b0071e45932351f0e762f65648a4b084ababc28a2ab319b05c549459b5e858354c9f026b775ba8cc93bb6afb28acf01edec65f637a342311c8962d006eec1ee2c1dffe75b27b9272ab21b6a34b17b3410d65299fcc5b24f0b4a53eafc4fba443b4ca084f44061d44e80961a0d8e4627a79142412d4f61bb763d5a1dc9b74e8b1c0ecf64c0fa0f60915ba4af25251563818313ecad9b663c0706158cf26cacfa8f178abc0564b8d20bfa1e41d5afe22f041a668c039da0caab718668489cf6da81986e91c0e02cb7d9d01ced06c3fa33a15b747aa733c9f04e336bf1450bb4578c679c02e0c352266cb66c12cbd69615b825f22e5df70a1119b7a432bb60837fba164bf762e63b49259a874e0017e863db70fbcffa6c7a934f0b51ae82bfa1908dc0dfbefcd22ac959ce6c845103c53b523666d890ac6d00a11e10490f3a5a04339a92c27252d33af77c711dbb4611668cb1dce29d967eacc5011ceb72334c4b26d9a96d9bec48ad84d8648a6fb9c37f883a1c2cab9118b5f00391cc333e28826edde8375953cf2f1a398bfd36a37e938509e914c19efa04fc35a3075ea149cc469cad140e73d6ec9ab8f2d1f1593a46d1963c3000a4c96be0b5363bbbfca5b5fbc70b74f9e26e525d14a2fdcce142fb8a227994a52e4cc3bc21d037f99b7a0afd7ea42c5c6af4c72f3945c6863eb86fdffdf7bb7a9df794508b8cc7a463c9f4df1b2157a8dcfb69bc67705ab3a351eae8b1387a61f78cb667236f12ca725f862044386bff2b781fbe715d52f6066f1f26ebf17ea1186de7f04b6f16f68c88a404c302e7efdf5de39b46c05cf86f37f6d9e1f1b2a19c4415dea7b881c1e69c6d54d3e1c7e7d466a8a78a7a3b5e4d5ad52caa47ce0901fa3bbf593b0ee523ef5854fd447ca77a6d2bf8b49869210b8d363628d4b6cdd853ff0276c9a95d58adca9a4280bb5c0bd55f4a1e8ec5102dd8eb132fc0b4b44178dc2f72ad015276e792f0cd0102d26ef550dfb7902595992ad15a676e6ff2c038d9d49b3affb197cb39a7197ca52867cc1237f924a2272e9b8b0ea8149d216d58abf72d023f89b2c5ebc3bee79dbdfe4ba64454c9b2e2095dbf613aacbbcbadb18118c245fdccfda8770139e26a1e44002d8d0dcd69ef4c715dcbc4471ce1c5d5298e2343ffbcd0a638aaad8fe87884642b9d1d435e9fec696a788b9848ce9165a3be37e8a7d9bc4549c2f3c9bb12c0b71e681046ebaa1096bd2faa9f682a13dfa178046e0346da4a5adf68734198abcc4d8b688746ec64d64f9bbc3ee37b6266dee17c6bd6c39ae39023ad56c2d89a4e7aba46abf9c6ce542ccae6da71727c851d9517b48fcbe52f3c02aca30d75d54df626181d224d8f909aab233f0beefdcdf4e8de99829f99086460a6c6f243bcd17b833ef22f4d75b5d9d779f6e7c15efb707150f781df054b5d64d217b16e0a27976fa32768d0b0f22b56d9a2a581346bc858209f92b506363821d9bba6823048b4da4304756176e3a008e7859e3179c4e159140471656e146fcbfef55eba1381e6a09bb9a81bf26a9eaff56ebfcb4406c5194c6c74ae976741121c797f5bad475b0d35e355764125e7b67b6d3c74d7bc4ff4195dd3c8a3b3e7c41334e7053fe9a365b24193562ffa35da82bfaaae2a2332e2bc02e497bee0a710be13f9ecfa30a6391f64bf29b0b86e1a4cacb7d71dc7f92210dc36ef5b951a426ab81ad5fc202d3b636750630339069ba7b8450ffc87abf4fa6715ab88f0918d807b3dedfdc7ef405792c4e9905b435a1cc66d76abffecfcaa6cd2aacc590961d40"}' [2023-09-07 17:14:02,519 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:14:02,768 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:14:02,768 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:14:02,769 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 17:14:02,771 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:14:02,772 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:14:02,773 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:14:02,773 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:14:02,773 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:14:02,774 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:14:02,778 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:14:02,782 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:14:02,792 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:14:02,825 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:14:02,904 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:14:02,906 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:14:02,906 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 17:14:02,945 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:14:02,946 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2500 [2023-09-07 17:14:02,947 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:14:02,947 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:14:02,948 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:14:02,948 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:14:02,949 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:14:02,962 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:14:02,964 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:14:02.964914 [2023-09-07 17:14:02,994 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:14:02,995 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:14:02,995 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:14:02,995 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:14:02,996 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:14:02,996 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:14:03,004 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:14:03,004 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:14:03,040 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:14:03,212 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:14:03,433 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:14:03,465 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:14:03,504 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:14:03,507 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:14:03,516 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:14:03,518 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694106843.4623892, 'te': 1694106843.4714663, 'initial_status': True} ====================================================================================================  [2023-09-07 17:14:03,520 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:14:03.520931 [2023-09-07 17:14:03,528 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:14:03,529 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:14:03,628 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:14:03,850 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:14:03,995 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:14:03,996 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2502 [2023-09-07 17:14:03,997 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:14:03,998 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:14:03,999 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:14:04,001 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:14:04,002 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:14:04,018 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:14:04,024 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:14:04.024426 [2023-09-07 17:14:04,122 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:14:04,271 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:14:04,272 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2487 [2023-09-07 17:14:04,273 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:14:04,273 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:14:04,274 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:14:04,274 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:14:04,275 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:14:04,285 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:14:04,289 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:14:04.289099 [2023-09-07 17:14:04,318 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:14:04,356 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:14:04,401 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:14:04,404 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:14:04,413 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:14:04,413 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694106844.3541522, 'te': 1694106844.3625278, 'initial_status': True} ====================================================================================================  [2023-09-07 17:14:04,415 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:14:04.415820 [2023-09-07 17:14:04,420 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:14:04,421 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:14:04,609 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:14:04,640 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:14:04,671 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:14:04,674 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:14:04,684 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:14:04,684 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694106844.6385639, 'te': 1694106844.6447933, 'initial_status': True} ====================================================================================================  [2023-09-07 17:14:04,687 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:14:04.687062 [2023-09-07 17:14:04,693 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:14:04,694 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:14:09,547 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:14:09,548 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:14:09,548 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:14:09,548 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:14:09,548 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:14:09,549 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:14:09,550 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:14:09,552 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 17:14:09,580 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:14:09,787 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:14:09,938 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:14:10,057 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:14:10,058 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2504 [2023-09-07 17:14:10,059 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:14:10,060 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:14:10,060 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:14:10,061 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:14:10,061 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:14:10,068 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:14:10,069 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:14:10.069934 [2023-09-07 17:14:10,212 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:14:10,241 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:14:10,272 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:14:10,274 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:14:10,281 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:14:10,282 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694106850.2392647, 'te': 1694106850.247091, 'initial_status': True} ====================================================================================================  [2023-09-07 17:14:10,283 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:14:10.283899 [2023-09-07 17:14:10,288 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:14:10,289 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:14:14,958 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:14:14,958 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:14:14,960 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:14:14.960604 [2023-09-07 17:14:14,966 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:14:22,763 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:14:22,763 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:14:22,765 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:14:22.765684 [2023-09-07 17:14:22,773 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:14:22,773 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:14:22,779 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:14:22,783 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694106862.7715843, 'te': 1694106862.783574, 'initial_status': True} ====================================================================================================  [2023-09-07 17:14:22,785 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:14:22.785328 [2023-09-07 17:14:22,790 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:14:22,790 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:14:24,081 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:14:24,082 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:14:24,084 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:14:24.084764 [2023-09-07 17:14:24,095 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:14:24,095 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:14:24,108 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:14:24,116 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694106864.092458, 'te': 1694106864.1159258, 'initial_status': True} ====================================================================================================  [2023-09-07 17:14:24,118 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:14:24.118951 [2023-09-07 17:14:24,123 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:14:24,124 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:14:24,135 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:14:24,135 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:14:24,138 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:14:24.138414 [2023-09-07 17:14:24,147 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:14:24,147 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:14:24,154 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:14:24,159 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694106864.1449559, 'te': 1694106864.1588976, 'initial_status': True} ====================================================================================================  [2023-09-07 17:14:24,161 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:14:24.161222 [2023-09-07 17:14:24,166 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:14:24,167 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:14:25,506 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 63 -> 65 [2023-09-07 17:14:29,683 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106864, schain_finish_ts: 1693742304 [2023-09-07 17:14:37,678 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106876, schain_finish_ts: 1692637668 [2023-09-07 17:14:38,747 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106876, schain_finish_ts: 0 [2023-09-07 17:14:41,725 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106876, schain_finish_ts: 1692377364 [2023-09-07 17:14:51,183 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106876, schain_finish_ts: 1694108304 [2023-09-07 17:14:53,463 INFO][1][MainProcess][MainThread] - core.schains.process_manager:57 - Process manager started [2023-09-07 17:14:54,612 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:14:54,612 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:14:54,612 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:14:54,612 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:14:54,612 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:14:54,613 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:14:54,615 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:14:54,619 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106888, schain_finish_ts: 1693742004 [2023-09-07 17:14:54,617 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:14:54,645 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:14:54,852 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:14:54,994 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:14:54,996 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:14:54,996 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:14:54,997 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:14:54,997 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:14:54,997 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:14:54,998 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 17:14:55,024 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:14:55,136 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:14:55,137 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2488 [2023-09-07 17:14:55,138 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:14:55,138 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:14:55,139 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:14:55,139 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:14:55,139 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:14:55,150 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:14:55,153 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:14:55.153089 [2023-09-07 17:14:55,237 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:14:55,361 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:14:55,395 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:14:55,433 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:14:55,436 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:14:55,444 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:14:55,444 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694106895.393936, 'te': 1694106895.4013796, 'initial_status': True} ====================================================================================================  [2023-09-07 17:14:55,446 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:14:55.446804 [2023-09-07 17:14:55,451 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:14:55,452 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:14:55,520 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:14:55,627 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:14:55,628 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2509 [2023-09-07 17:14:55,629 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:14:55,629 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:14:55,629 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:14:55,630 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:14:55,630 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:14:55,636 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:14:55,638 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:14:55.638580 [2023-09-07 17:14:55,796 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:14:55,835 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:14:55,880 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:14:55,884 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:14:55,901 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:14:55,902 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694106895.8326545, 'te': 1694106895.841086, 'initial_status': True} ====================================================================================================  [2023-09-07 17:14:55,905 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:14:55.905543 [2023-09-07 17:14:55,912 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:14:55,914 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:14:59,722 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:14:59,723 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:14:59,725 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:14:59,726 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:14:59,727 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:14:59,728 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:14:59,731 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:14:59,731 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 17:14:59,778 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:14:59,779 INFO][1][MainProcess][MainThread] - core.schains.notifications:35 - Node account has 406909841189497969 WEI [2023-09-07 17:14:59,780 INFO][1][MainProcess][MainThread] - tools.notifications.messages:188 - Saving new balance state 3010 1 [2023-09-07 17:14:59,781 INFO][1][MainProcess][MainThread] - core.schains.process_manager:95 - Fetching schains to monitor... [2023-09-07 17:14:59,891 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106888, schain_finish_ts: 1693412520 [2023-09-07 17:15:00,002 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:15:00,205 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:15:00,349 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:15:00,350 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2503 [2023-09-07 17:15:00,351 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:15:00,351 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:15:00,351 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:15:00,351 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:15:00,352 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:15:00,359 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:15:00,362 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:15:00.362240 [2023-09-07 17:15:00,520 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:15:00,552 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:15:00,590 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:15:00,593 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:15:00,606 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:15:00,606 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694106900.550117, 'te': 1694106900.5573325, 'initial_status': True} ====================================================================================================  [2023-09-07 17:15:00,608 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:15:00.608699 [2023-09-07 17:15:00,613 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:15:00,614 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:15:05,199 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:15:05,200 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:15:05,200 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 17:15:07,893 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:15:07,894 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:15:07,895 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:15:08,105 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:15:08,105 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:15:08,105 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:15:08,106 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:15:08,106 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:15:08,106 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 17:15:08,132 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:15:08,533 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:15:08,679 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:15:08,817 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:15:08,818 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2501 [2023-09-07 17:15:08,819 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:15:08,819 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:15:08,820 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:15:08,820 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:15:08,820 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:15:08,827 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:15:08,831 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:15:08.831266 [2023-09-07 17:15:08,987 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:15:09,021 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:15:09,049 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:15:09,051 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:15:09,058 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:15:09,059 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694106909.019567, 'te': 1694106909.026407, 'initial_status': True} ====================================================================================================  [2023-09-07 17:15:09,060 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:15:09.060839 [2023-09-07 17:15:09,065 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:15:09,066 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:15:27,122 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:15:27,122 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:15:27,124 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:15:27.124923 [2023-09-07 17:15:27,132 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:15:27,133 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:15:27,138 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:15:27,143 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694106927.1302907, 'te': 1694106927.1431985, 'initial_status': True} ====================================================================================================  [2023-09-07 17:15:27,145 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:15:27.145481 [2023-09-07 17:15:27,149 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:15:27,149 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:15:27,379 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:15:27,379 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:15:27,380 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:15:27,380 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:15:27,380 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:15:27,381 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:15:27,382 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:15:27,384 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:15:27,412 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:15:27,471 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:15:27,471 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:15:27,474 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:15:27.474535 [2023-09-07 17:15:27,482 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:15:27,483 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:15:27,489 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:15:27,495 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694106927.4814768, 'te': 1694106927.4955218, 'initial_status': True} ====================================================================================================  [2023-09-07 17:15:27,497 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:15:27.497467 [2023-09-07 17:15:27,502 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:15:27,502 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:15:27,754 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:15:27,940 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:15:28,063 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:15:28,064 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2505 [2023-09-07 17:15:28,065 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:15:28,066 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:15:28,066 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:15:28,066 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:15:28,067 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:15:28,073 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:15:28,075 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:15:28.075830 [2023-09-07 17:15:28,229 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:15:28,255 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:15:28,284 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:15:28,287 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:15:28,293 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:15:28,294 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694106928.2540457, 'te': 1694106928.2600477, 'initial_status': True} ====================================================================================================  [2023-09-07 17:15:28,295 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:15:28.295726 [2023-09-07 17:15:28,299 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:15:28,300 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:15:30,789 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 23 -> 24 [2023-09-07 17:15:38,075 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106888, schain_finish_ts: 1693742304 [2023-09-07 17:15:43,720 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106936, schain_finish_ts: 1692637668 [2023-09-07 17:15:43,924 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106936, schain_finish_ts: 1694108304 [2023-09-07 17:15:45,440 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106936, schain_finish_ts: 0 [2023-09-07 17:15:47,954 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:15:47,954 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:15:47,956 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:15:47.956458 [2023-09-07 17:15:47,962 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:15:47,962 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:15:47,968 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:15:47,972 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694106947.9613452, 'te': 1694106947.972352, 'initial_status': True} ====================================================================================================  [2023-09-07 17:15:47,974 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:15:47.974057 [2023-09-07 17:15:47,978 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:15:47,978 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:15:55,958 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:15:55,958 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:15:55,959 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:15:55,959 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:15:55,959 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:15:55,959 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:15:55,962 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:15:55,963 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:15:55,990 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:15:56,342 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:15:56,520 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:15:56,652 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:15:56,653 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2504 [2023-09-07 17:15:56,654 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:15:56,654 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:15:56,655 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:15:56,655 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:15:56,655 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:15:56,661 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:15:56,663 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:15:56.663420 [2023-09-07 17:15:56,797 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:15:56,837 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:15:56,866 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:15:56,868 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:15:56,877 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:15:56,878 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694106956.8347878, 'te': 1694106956.8428276, 'initial_status': True} ====================================================================================================  [2023-09-07 17:15:56,881 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:15:56.881158 [2023-09-07 17:15:56,885 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:15:56,886 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:15:58,961 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106948, schain_finish_ts: 1692377364 [2023-09-07 17:15:59,050 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:15:59,051 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:15:59,051 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:16:00,375 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106948, schain_finish_ts: 1693412520 [2023-09-07 17:16:07,007 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:16:07,007 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:16:07,007 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:16:07,008 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:16:07,008 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:16:07,009 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:16:07,011 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:16:07,014 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 17:16:07,043 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:16:07,272 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:16:07,426 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:16:07,715 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:16:07,716 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2489 [2023-09-07 17:16:07,716 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:16:07,717 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:16:07,717 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:16:07,717 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:16:07,717 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:16:07,723 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:16:07,725 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:16:07.725677 [2023-09-07 17:16:07,861 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:16:07,889 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:16:07,918 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:16:07,920 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:16:07,927 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:16:07,927 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694106967.8882134, 'te': 1694106967.8949888, 'initial_status': True} ====================================================================================================  [2023-09-07 17:16:07,929 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:16:07.929254 [2023-09-07 17:16:07,933 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:16:07,934 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:16:15,666 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:16:15,667 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:16:15,667 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 17:16:18,216 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:16:18,217 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:16:18,217 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:16:18,217 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:16:18,217 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:16:18,218 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:16:18,241 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:16:18,416 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:16:18,544 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:16:18,645 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:16:18,646 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2510 [2023-09-07 17:16:18,646 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:16:18,647 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:16:18,647 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:16:18,647 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:16:18,648 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:16:18,654 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:16:18,656 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:16:18.656293 [2023-09-07 17:16:18,930 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:16:18,957 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:16:18,986 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:16:18,988 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:16:18,995 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:16:18,996 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694106978.9558315, 'te': 1694106978.9619272, 'initial_status': True} ====================================================================================================  [2023-09-07 17:16:18,998 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:16:18.998545 [2023-09-07 17:16:19,003 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:16:19,004 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:16:25,771 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106972, schain_finish_ts: 1693742004 [2023-09-07 17:16:32,364 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:16:32,365 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:16:32,365 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:16:32,365 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:16:32,366 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:16:32,366 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:16:32,368 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:16:32,370 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 17:16:32,416 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:16:32,472 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106984, schain_finish_ts: 1694108304 [2023-09-07 17:16:32,758 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:16:32,980 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:16:33,104 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:16:33,105 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2506 [2023-09-07 17:16:33,106 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:16:33,106 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:16:33,106 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:16:33,107 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:16:33,107 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:16:33,114 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:16:33,116 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:16:33.116415 [2023-09-07 17:16:33,482 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:16:33,511 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:16:33,543 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:16:33,545 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:16:33,553 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:16:33,553 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694106993.5102086, 'te': 1694106993.5164483, 'initial_status': True} ====================================================================================================  [2023-09-07 17:16:33,555 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:16:33.555413 [2023-09-07 17:16:33,559 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:16:33,560 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:16:37,902 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:16:37,902 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:16:37,904 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:16:37.904787 [2023-09-07 17:16:37,912 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:16:37,912 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:16:37,918 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:16:37,923 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694106997.9104407, 'te': 1694106997.9231634, 'initial_status': True} ====================================================================================================  [2023-09-07 17:16:37,925 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:16:37.925488 [2023-09-07 17:16:37,931 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:16:37,931 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:16:37,933 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.transactions.tools:190 - Tx broadcast completed after 1/3 retries [2023-09-07 17:16:37,934 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:230 - sChain: free-basic-rosy-SKALE. Everything is sent from 14 node [2023-09-07 17:16:39,479 INFO][1][MainProcess][MainThread] - core.schains.process_manager:109 - Get leaving_history for node ... [2023-09-07 17:16:40,740 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:104 - sChain free-basic-rosy-SKALE: trying to receive broadcasted data,1416 seconds left [2023-09-07 17:16:43,844 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106996, schain_finish_ts: 1692377364 [2023-09-07 17:16:44,181 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:16:44,183 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:16:44,185 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:16:44,187 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:16:44,188 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:16:44,190 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:16:44,237 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:16:44,431 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:16:44,563 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:16:44,689 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:16:44,690 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2502 [2023-09-07 17:16:44,690 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:16:44,691 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:16:44,691 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:16:44,691 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:16:44,692 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:16:44,699 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:16:44,702 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:16:44.702682 [2023-09-07 17:16:45,018 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:16:45,049 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:16:45,090 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:16:45,093 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:16:45,100 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:16:45,100 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694107005.0474591, 'te': 1694107005.0537128, 'initial_status': True} ====================================================================================================  [2023-09-07 17:16:45,101 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:16:45.101878 [2023-09-07 17:16:45,105 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:16:45,106 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:16:46,259 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:16:46,261 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:16:46,262 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:16:48,449 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 17:16:48,473 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.broadcast_filter:94 - sChain 0x000002bb9043962a56bacebbde03191f32a67a4b888426ac09f83548ae92e235: Parsing broadcast events from 9653867 block to 9653890 block [2023-09-07 17:16:48,581 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694106996, schain_finish_ts: 1692637668 [2023-09-07 17:16:48,625 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:16:48,625 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:16:48,625 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:16:48,626 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:16:48,791 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:16:48,791 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:16:48,793 INFO][8][curly-red-alterf][T_0] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 17:16:48,798 ERROR][8][curly-red-alterf][T_0] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:16:48,799 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:16:54,501 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:16:54,501 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:16:54,503 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:16:54.503592 [2023-09-07 17:16:54,510 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:16:54,510 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:16:54,516 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:16:54,520 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694107014.5089588, 'te': 1694107014.520227, 'initial_status': True} ====================================================================================================  [2023-09-07 17:16:54,522 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:16:54.522313 [2023-09-07 17:16:54,526 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:16:54,526 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:16:57,849 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:16:57,850 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:16:57,852 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:16:57.852378 [2023-09-07 17:16:57,859 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:16:57,860 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:16:57,865 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:16:57,870 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694107017.8582413, 'te': 1694107017.8699367, 'initial_status': True} ====================================================================================================  [2023-09-07 17:16:57,872 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:16:57.872774 [2023-09-07 17:16:57,877 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:16:57,877 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:16:57,885 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107008, schain_finish_ts: 1692377616 [2023-09-07 17:17:01,429 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:17:01,430 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:17:01,430 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:17:01,431 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:17:01,431 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:17:01,431 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:17:01,435 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:17:01,438 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:17:01,463 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:17:01,473 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 9 -> 64 [2023-09-07 17:17:01,532 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107020, schain_finish_ts: 0 [2023-09-07 17:17:01,649 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:17:01,779 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:17:01,888 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:17:01,888 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:17:01,889 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:17:01,889 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:17:01,889 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:17:01,889 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:17:01,895 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:17:01,896 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 17:17:01,906 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:17:01,908 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2490 [2023-09-07 17:17:01,909 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:17:01,909 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:17:01,909 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:17:01,910 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:17:01,910 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:17:01,919 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:17:01,922 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:17:01.922050 [2023-09-07 17:17:01,928 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:17:02,080 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:17:02,108 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:17:02,143 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:17:02,145 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:17:02,151 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:17:02,152 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694107022.106554, 'te': 1694107022.1162114, 'initial_status': True} ====================================================================================================  [2023-09-07 17:17:02,153 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:17:02.153606 [2023-09-07 17:17:02,157 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:17:02,158 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:17:02,215 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:17:02,352 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:17:02,450 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:17:02,451 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2505 [2023-09-07 17:17:02,452 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:17:02,452 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:17:02,452 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:17:02,452 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:17:02,453 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:17:02,458 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:17:02,460 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:17:02.460734 [2023-09-07 17:17:02,706 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:17:02,735 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:17:02,764 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:17:02,765 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:17:02,772 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:17:02,772 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694107022.7330327, 'te': 1694107022.7419643, 'initial_status': True} ====================================================================================================  [2023-09-07 17:17:02,774 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:17:02.774398 [2023-09-07 17:17:02,778 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:17:02,779 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:17:03,139 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107020, schain_finish_ts: 1693742304 [2023-09-07 17:17:05,110 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107020, schain_finish_ts: 0 [2023-09-07 17:17:07,853 INFO][1][MainProcess][MainThread] - core.schains.process_manager:117 - Got leaving sChains for the node: [] [2023-09-07 17:17:07,854 INFO][1][MainProcess][MainThread] - core.schains.process_manager:101 -  ==================================================================================================== Monitoring sChains Node ID: 16 sChains on node: [{'name': 'portly-passionate-sirius', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666281384, 'startBlock': 7803380, 'deposit': 0, 'index': 1, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x0000017723ca2', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'curly-red-alterf', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1676058936, 'startBlock': 8469646, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000001fe6cdfa', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'howling-hoedus', 'mainnetOwner': '0x1026467C91fdA40aCc7e6935d1E52f67274FD669', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666181400, 'startBlock': 7796660, 'deposit': 0, 'index': 0, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0xe5a600f4c2608', 'multitransactionMode': False, 'thresholdEncryption': False, 'active': True}, {'name': 'green-giddy-denebola', 'mainnetOwner': '0x8e835d5d9E28ffD259E8Fa8988Cacfc333A582B4', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674565164, 'startBlock': 8368049, 'deposit': 0, 'index': 4, 'generation': 1, 'originator': '0xd2001DAb6898127Be2F167B548691C87251D13C3', 'chainId': '0x00000585eb4b1', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'wan-tasty-kitalpha', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674568392, 'startBlock': 8368271, 'deposit': 0, 'index': 5, 'generation': 1, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x00000152e7be9', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'free-basic-rosy-SKALE', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1693996920, 'startBlock': 9646443, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000002bb90439', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'submissive-well-off-rotanev', 'mainnetOwner': '0x487360568fE5Ee4AE5c75782D5F63e9935B0E512', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1692361104, 'startBlock': 9539053, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000003791fa10', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}] Number of sChains on node: 7 Empty sChain structs: 0 ====================================================================================================  [2023-09-07 17:17:08,098 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107020, schain_finish_ts: 1693412520 [2023-09-07 17:17:10,349 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:17:10,349 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:17:10,349 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 17:17:10,447 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:17:10,447 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain portly-passionate-sirius - Process is running: PID = 7 [2023-09-07 17:17:14,036 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:17:14,038 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain curly-red-alterf - Process is running: PID = 8 [2023-09-07 17:17:14,075 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:17:14,075 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:17:14,075 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:17:14,075 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:17:14,076 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:17:14,076 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 17:17:14,098 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:17:14,219 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:17:14,219 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:17:14,223 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:17:14.223125 [2023-09-07 17:17:14,233 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:17:14,233 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:17:14,241 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:17:14,246 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694107034.2305772, 'te': 1694107034.2459183, 'initial_status': True} ====================================================================================================  [2023-09-07 17:17:14,248 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:17:14.248476 [2023-09-07 17:17:14,253 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:17:14,253 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:17:14,385 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:17:14,544 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:17:14,661 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:17:14,662 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2511 [2023-09-07 17:17:14,662 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:17:14,662 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:17:14,663 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:17:14,663 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:17:14,663 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:17:14,670 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:17:14,672 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:17:14.672462 [2023-09-07 17:17:14,839 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:17:14,874 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:17:14,901 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:17:14,902 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:17:14,909 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:17:14,910 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694107034.8725734, 'te': 1694107034.8787184, 'initial_status': True} ====================================================================================================  [2023-09-07 17:17:14,911 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:17:14.911550 [2023-09-07 17:17:14,915 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:17:14,916 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:17:19,287 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:17:19,287 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain howling-hoedus - Process is running: PID = 9 [2023-09-07 17:17:22,612 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:17:22,615 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:17:22,615 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:17:22,617 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:17:22,619 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:17:22,621 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:17:22,625 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:17:22,628 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 17:17:22,657 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:17:22,796 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:17:22,796 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain green-giddy-denebola - Process is running: PID = 10 [2023-09-07 17:17:22,886 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:17:23,076 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:17:23,210 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:17:23,212 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2507 [2023-09-07 17:17:23,213 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:17:23,213 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:17:23,213 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:17:23,213 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:17:23,214 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:17:23,219 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:17:23,221 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:17:23.221703 [2023-09-07 17:17:23,361 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:17:23,396 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:17:23,429 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:17:23,431 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:17:23,438 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:17:23,438 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694107043.392957, 'te': 1694107043.4052327, 'initial_status': True} ====================================================================================================  [2023-09-07 17:17:23,440 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:17:23.440815 [2023-09-07 17:17:23,447 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:17:23,448 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:17:26,170 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:17:26,171 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain wan-tasty-kitalpha - Process is running: PID = 11 [2023-09-07 17:17:27,180 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:17:27,180 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:17:27,182 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:17:27.182029 [2023-09-07 17:17:27,189 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:17:27,189 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:17:27,195 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:17:27,199 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694107047.1882033, 'te': 1694107047.1995173, 'initial_status': True} ====================================================================================================  [2023-09-07 17:17:27,201 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:17:27.201253 [2023-09-07 17:17:27,206 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:17:27,206 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:17:28,011 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107020, schain_finish_ts: 1693742004 [2023-09-07 17:17:29,763 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:17:29,764 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain free-basic-rosy-SKALE - Process is running: PID = 30622 [2023-09-07 17:17:30,780 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:17:30,780 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:17:30,782 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:17:30.782358 [2023-09-07 17:17:30,792 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:17:30,792 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:17:30,800 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:17:30,805 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694107050.790833, 'te': 1694107050.8055618, 'initial_status': True} ====================================================================================================  [2023-09-07 17:17:30,807 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:17:30.807604 [2023-09-07 17:17:30,812 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:17:30,812 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:17:31,860 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107044, schain_finish_ts: 1694108304 [2023-09-07 17:17:33,159 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:17:33,159 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain submissive-well-off-rotanev - Process is running: PID = 24912 [2023-09-07 17:17:33,160 INFO][1][MainProcess][MainThread] - core.schains.process_manager:88 - Process manager procedure finished [2023-09-07 17:17:33,160 INFO][1][MainProcess][MainThread] - __main__:65 - Sleeping for 90s after run_process_manager [2023-09-07 17:17:40,605 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:17:40,605 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:17:40,606 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 17:17:41,480 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107044, schain_finish_ts: 1692377364 [2023-09-07 17:17:42,196 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107044, schain_finish_ts: 1692637668 [2023-09-07 17:17:45,655 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:17:45,655 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:17:45,655 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:17:45,656 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 17:17:45,656 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 17:17:45,656 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:17:45,660 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:17:45,661 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 17:17:45,723 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:17:45,949 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:17:46,122 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:17:46,256 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:17:46,257 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2503 [2023-09-07 17:17:46,257 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:17:46,258 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:17:46,258 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:17:46,258 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:17:46,258 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:17:46,265 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:17:46,267 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:17:46.267105 [2023-09-07 17:17:46,416 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:17:46,444 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:17:46,473 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:17:46,475 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:17:46,487 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:17:46,487 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694107066.4428217, 'te': 1694107066.4495404, 'initial_status': True} ====================================================================================================  [2023-09-07 17:17:46,489 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:17:46.489148 [2023-09-07 17:17:46,495 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:17:46,498 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:17:46,739 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 35 -> 16 [2023-09-07 17:17:55,879 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:17:55,879 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:17:55,879 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:17:55,879 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:17:55,879 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:17:55,880 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:17:55,882 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:17:55,888 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 17:17:55,918 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:17:56,144 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:17:56,447 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:17:56,551 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:17:56,552 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2491 [2023-09-07 17:17:56,553 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:17:56,553 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:17:56,553 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:17:56,553 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:17:56,554 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:17:56,560 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:17:56,562 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:17:56.562270 [2023-09-07 17:17:56,702 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:17:56,727 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:17:56,754 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:17:56,756 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:17:56,765 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:17:56,765 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694107076.7256951, 'te': 1694107076.7318127, 'initial_status': True} ====================================================================================================  [2023-09-07 17:17:56,767 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:17:56.767126 [2023-09-07 17:17:56,771 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:17:56,772 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:18:00,982 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107068, schain_finish_ts: 0 [2023-09-07 17:18:05,269 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:18:05,270 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:18:05,270 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:18:05,270 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:18:05,270 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:18:05,271 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:18:05,273 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:18:05,275 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:18:05,304 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:18:05,501 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:18:05,640 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:18:05,748 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:18:05,749 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2506 [2023-09-07 17:18:05,749 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:18:05,749 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:18:05,750 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:18:05,750 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:18:05,750 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:18:05,759 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:18:05,761 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:18:05.761799 [2023-09-07 17:18:05,917 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:18:05,949 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:18:05,998 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:18:06,001 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:18:06,011 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:18:06,012 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694107085.9472246, 'te': 1694107085.9617095, 'initial_status': True} ====================================================================================================  [2023-09-07 17:18:06,014 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:18:06.014158 [2023-09-07 17:18:06,021 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:18:06,022 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:18:15,325 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107068, schain_finish_ts: 1693742304 [2023-09-07 17:18:19,203 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107092, schain_finish_ts: 1693412520 [2023-09-07 17:18:19,368 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:18:19,368 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:18:19,370 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:18:19.370381 [2023-09-07 17:18:19,380 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:18:19,380 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:18:19,387 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:18:19,396 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694107099.3785403, 'te': 1694107099.3951728, 'initial_status': True} ====================================================================================================  [2023-09-07 17:18:19,398 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:18:19.398573 [2023-09-07 17:18:19,404 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:18:19,404 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:18:26,712 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:18:26,712 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:18:26,712 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:18:29,054 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107104, schain_finish_ts: 1694108304 [2023-09-07 17:18:29,357 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:18:29,357 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:18:29,359 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:18:29.359229 [2023-09-07 17:18:29,364 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:18:31,307 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:18:31,308 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:18:31,308 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:18:31,308 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:18:31,308 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:18:31,309 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:18:31,310 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:18:31,313 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 17:18:31,345 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:18:31,560 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:18:31,797 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:18:31,900 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:18:31,901 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2508 [2023-09-07 17:18:31,902 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:18:31,902 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:18:31,902 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:18:31,902 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:18:31,903 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:18:31,909 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:18:31,911 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:18:31.911387 [2023-09-07 17:18:32,082 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:18:32,116 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:18:32,148 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:18:32,150 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:18:32,158 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:18:32,158 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694107112.114669, 'te': 1694107112.1206713, 'initial_status': True} ====================================================================================================  [2023-09-07 17:18:32,160 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:18:32.160110 [2023-09-07 17:18:32,164 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:18:32,164 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:18:35,741 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107104, schain_finish_ts: 1693742004 [2023-09-07 17:18:37,461 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:18:37,463 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:18:37,466 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:18:37.466702 [2023-09-07 17:18:37,476 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:18:37,476 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:18:37,483 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:18:37,489 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694107117.473552, 'te': 1694107117.4890268, 'initial_status': True} ====================================================================================================  [2023-09-07 17:18:37,492 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:18:37.492800 [2023-09-07 17:18:37,499 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:18:37,500 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:18:41,894 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:18:41,894 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:18:41,895 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:18:41,895 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:18:41,895 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:18:41,896 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:18:41,919 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:18:42,001 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107116, schain_finish_ts: 1692377364 [2023-09-07 17:18:42,126 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:18:42,420 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:18:42,534 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:18:42,535 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2512 [2023-09-07 17:18:42,536 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:18:42,536 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:18:42,537 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:18:42,537 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:18:42,538 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:18:42,545 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:18:42,547 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:18:42.547630 [2023-09-07 17:18:42,766 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:18:42,793 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:18:42,825 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:18:42,827 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:18:42,834 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:18:42,834 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694107122.7919202, 'te': 1694107122.799073, 'initial_status': True} ====================================================================================================  [2023-09-07 17:18:42,836 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:18:42.836282 [2023-09-07 17:18:42,841 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:18:42,841 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:18:47,151 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:18:47,151 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:18:47,151 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 17:18:52,657 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107128, schain_finish_ts: 1692637668 [2023-09-07 17:18:55,017 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:18:55,017 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:18:55,017 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:18:55,017 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:18:55,017 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:18:55,018 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:18:55,048 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:18:55,173 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 17:18:55,173 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 17:18:55,264 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:18:55,422 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:18:55,527 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:18:55,528 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2504 [2023-09-07 17:18:55,529 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:18:55,529 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:18:55,529 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:18:55,529 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:18:55,530 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:18:55,536 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:18:55,538 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:18:55.538175 [2023-09-07 17:18:55,792 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:18:55,822 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:18:55,850 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:18:55,852 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:18:55,859 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:18:55,860 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694107135.820559, 'te': 1694107135.8277721, 'initial_status': True} ====================================================================================================  [2023-09-07 17:18:55,861 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:18:55.861807 [2023-09-07 17:18:55,866 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:18:55,867 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:18:58,180 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:18:58,181 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:18:58,181 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:18:58,181 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:18:58,181 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:18:58,182 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:18:58,187 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:18:58,188 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 17:18:58,209 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:18:58,417 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:18:58,550 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:18:58,671 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:18:58,673 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2492 [2023-09-07 17:18:58,674 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:18:58,674 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:18:58,677 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:18:58,678 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:18:58,678 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:18:58,699 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:18:58,735 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:18:58.735728 [2023-09-07 17:18:58,965 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:18:58,991 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:18:59,021 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:18:59,023 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:18:59,030 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:18:59,030 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694107138.9896605, 'te': 1694107138.995776, 'initial_status': True} ====================================================================================================  [2023-09-07 17:18:59,032 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:18:59.032616 [2023-09-07 17:18:59,036 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:18:59,037 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:19:03,253 INFO][1][MainProcess][MainThread] - core.schains.cleaner:65 - Cleaner process started [2023-09-07 17:19:03,282 INFO][30673][cleaner][MainThread] - core.schains.cleaner:110 - Cleaner procedure started. [2023-09-07 17:19:11,716 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107140, schain_finish_ts: 0 [2023-09-07 17:19:22,404 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107152, schain_finish_ts: 1693412520 [2023-09-07 17:19:28,032 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:19:28,032 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:19:28,032 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:19:28,033 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:19:28,033 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:19:28,033 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:19:28,035 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:19:28,037 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 17:19:28,059 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:19:28,248 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:19:28,382 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:19:28,480 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:19:28,481 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2507 [2023-09-07 17:19:28,482 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:19:28,482 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:19:28,483 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:19:28,483 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:19:28,483 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:19:28,489 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:19:28,491 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:19:28.491191 [2023-09-07 17:19:28,743 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:19:28,771 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:19:28,803 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:19:28,804 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:19:28,812 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:19:28,813 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694107168.7686102, 'te': 1694107168.778586, 'initial_status': True} ====================================================================================================  [2023-09-07 17:19:28,815 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:19:28.815175 [2023-09-07 17:19:28,821 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:19:28,822 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:19:33,971 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107152, schain_finish_ts: 1693742004 [2023-09-07 17:19:34,247 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107152, schain_finish_ts: 1692377364 [2023-09-07 17:19:35,024 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:19:35,024 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:19:35,026 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:19:35.026860 [2023-09-07 17:19:35,034 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:19:35,034 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:19:35,039 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:19:35,044 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694107175.0327754, 'te': 1694107175.0444381, 'initial_status': True} ====================================================================================================  [2023-09-07 17:19:35,046 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:19:35.046284 [2023-09-07 17:19:35,050 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:19:35,050 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:19:40,450 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:19:40,450 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:19:40,453 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:19:40.453018 [2023-09-07 17:19:40,460 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:19:40,460 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:19:40,465 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:19:40,469 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694107180.4588006, 'te': 1694107180.4696014, 'initial_status': True} ====================================================================================================  [2023-09-07 17:19:40,471 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:19:40.471384 [2023-09-07 17:19:40,476 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:19:40,477 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:19:41,764 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 55 -> 63 [2023-09-07 17:19:42,734 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:19:42,735 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:19:42,735 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:19:42,735 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:19:42,736 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:19:42,736 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:19:42,748 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:19:42,749 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 17:19:42,783 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:19:43,002 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107176, schain_finish_ts: 1694108304 [2023-09-07 17:19:43,073 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:19:43,223 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:19:43,345 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:19:43,346 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2509 [2023-09-07 17:19:43,347 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:19:43,348 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:19:43,348 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:19:43,348 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:19:43,349 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:19:43,359 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:19:43,362 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:19:43.362747 [2023-09-07 17:19:43,584 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:19:43,618 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:19:43,657 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:19:43,660 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:19:43,668 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:19:43,669 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694107183.61709, 'te': 1694107183.6250172, 'initial_status': True} ====================================================================================================  [2023-09-07 17:19:43,671 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:19:43.671620 [2023-09-07 17:19:43,675 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:19:43,676 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:19:44,325 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:19:44,325 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:19:44,326 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 17:19:50,122 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107176, schain_finish_ts: 1693742304 [2023-09-07 17:19:53,629 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:19:53,630 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:19:53,630 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:19:53,630 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:19:53,630 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:19:53,631 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 17:19:53,661 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:19:53,700 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:19:53,701 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:19:53,701 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 17:19:53,742 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:19:53,743 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:19:53,743 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:19:53,743 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:19:53,743 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:19:53,744 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:19:53,746 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:19:53,749 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:19:53,784 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:19:53,857 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:19:54,017 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:19:54,285 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:19:54,420 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:19:54,421 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2505 [2023-09-07 17:19:54,422 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:19:54,422 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:19:54,423 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:19:54,423 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:19:54,423 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:19:54,431 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:19:54,434 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:19:54.434414 [2023-09-07 17:19:54,618 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:19:54,746 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:19:54,747 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2493 [2023-09-07 17:19:54,748 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:19:54,748 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:19:54,749 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:19:54,749 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:19:54,749 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:19:54,757 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:19:54,759 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:19:54.759299 [2023-09-07 17:19:54,799 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:19:54,832 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:19:54,863 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:19:54,865 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:19:54,872 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:19:54,873 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694107194.8294263, 'te': 1694107194.8372462, 'initial_status': True} ====================================================================================================  [2023-09-07 17:19:54,875 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:19:54.875139 [2023-09-07 17:19:54,879 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:19:54,880 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:19:54,937 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:19:54,963 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:19:54,990 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:19:54,992 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:19:54,999 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:19:54,999 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694107194.9614322, 'te': 1694107194.967479, 'initial_status': True} ====================================================================================================  [2023-09-07 17:19:55,001 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:19:55.001431 [2023-09-07 17:19:55,007 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:19:55,008 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:19:56,602 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:19:56,602 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:19:56,604 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:19:56.604922 [2023-09-07 17:19:56,611 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:19:56,612 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:19:56,617 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:19:56,622 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694107196.6104085, 'te': 1694107196.6218529, 'initial_status': True} ====================================================================================================  [2023-09-07 17:19:56,623 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:19:56.623666 [2023-09-07 17:19:56,627 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:19:56,627 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:19:57,114 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:19:57,114 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:19:57,116 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:19:57.116571 [2023-09-07 17:19:57,123 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:19:57,123 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:19:57,129 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:19:57,133 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694107197.1216326, 'te': 1694107197.1332507, 'initial_status': True} ====================================================================================================  [2023-09-07 17:19:57,135 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:19:57.135123 [2023-09-07 17:19:57,139 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:19:57,140 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:19:57,161 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:19:57,161 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:19:57,161 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:19:57,161 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:19:57,162 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:19:57,162 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:19:57,183 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:19:57,390 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:19:57,547 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:19:57,658 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:19:57,659 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2513 [2023-09-07 17:19:57,660 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:19:57,660 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:19:57,660 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:19:57,660 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:19:57,661 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:19:57,666 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:19:57,668 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:19:57.668758 [2023-09-07 17:19:57,800 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:19:57,827 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:19:57,859 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:19:57,861 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:19:57,868 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:19:57,869 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694107197.8256948, 'te': 1694107197.832436, 'initial_status': True} ====================================================================================================  [2023-09-07 17:19:57,870 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:19:57.870675 [2023-09-07 17:19:57,876 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:19:57,876 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:20:06,023 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 6 [2023-09-07 17:20:06,024 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '14d1f433972b93250246e818df1ee153d8a533050d91ecc4673bc02ff69a8b1309a9c7c928d673b8a3bc40dea862a4879c094829db2b803b6bb550b5d5f65a0502e82c96bf1c7e5c470ae76a9ebd3985807e2c255edb390f12139b77c590450c282100c66c5ecda3d06a6dc2e0a49d3dbf5444c0f1f0aaac27f3af2a1dc8e38c194e9462a48fe4eddc00244529e2b5a22c6f67b64fe9a2b59e4cda7bcefb1ca30e80f923af6c954206d96bb6c9ff1c23697e30ee488dce6ded85df59a1631dd92c1b337abd0d0ce2f41a34663cf3525d5605726ce6c585a1dd81ee15231f8fa70a1f031840ecf561064d502e6e1590f5b6f6767dc9208224aa15b4301cd5622926debfc747b2b2e0976b572d624cbc83678298410ea2424fc9de19a807ffef1615d536f89a383bdf237e1e623365cfb7c648fb6069aaaf0c64108e3893e8b82d106fcd96672ebb0daa75db49bdd9010768abe66ca57e8a7cce2c286557ea7191032803c861aca5d6d579bd2125c9e45cad9c7cb245afa91cf36ff9761f1dc2a80a960baffe38277adcd4a72b6f7ff8e82157a9124b7e90aee8ec46439e31ba2e2ef19f9b71d2eac8c1e9e7ad8c96f3c0f04669f7fcf311cbc7e1e5384301b354246ff5da88a8dd2a1c4a18de67633de80fdcd75fdc012527797a74ccc70adefb1ea70a7be0fea831e31b6f59fe5004555115849ae17c7f63809666dd2bb04f6114fb1b97016ca5263aefab6c6656b1e448f493048b7ea5c83fe172427b127a67298064c10ba20be01610caeeccf16a59fe36d4d212888aaabd69ec583935177c074388c698272d44544d2f2d901661d61b73c923471f9e4c5050975273f8223b2c347f25f23357ada83565fa9b66a070a381d1cc95a2b8f9add82817d1cca67c15f89504f684eb0c0b9aa8c4a792bbed6d1fa4fcb5d8ec4aef48e3da6bd67e4b19589df2ec529210e15d1c71fcd26bfde11e5a747ac1564ff102b7ebb33615061d4f20533fed72d5ba91622b6b088c37c7715d29bf7240a157e6e187a27208271d8b30534023c31487eb0784d63ad3274f8774529d4025cf377128e73b0cf3ff0e8fe49a211c33e349a3a4d67672d5c11f2844b0bc5826fc4890e5fdecf242652843a2ca018c29a2aebc96b3f48f91dd17caf1ca859c83b056c4fac21c36856e240e964ac0586cb5d796a968e917940c1d523be2990a8d4c19a578de0ab21db005628bd76ff3aa66c414832562191e675e26751acc9d543d8518950de846337a22f4e413b0b2b51a3cd4fcb83ea2acc41e34e0fbd839a3b56a0b7a1b38efb63a290711d2f02bf6e55f155c17a92ae336c2b88edc99bee81ebc2b8d29161250d514d04792e6562dee7ce3fc3dbd4882783133375d5f559574ad3adcc000b88c0522163af0262bf8313fed2c96fe65b807d0189c12aaf8b5634f4336755cf58c500a755ba7cca9ddb8cac0f0873162ce66067e561716f470ae0c44ee42d93fb92e2cd27b70b0c3c6bdbf504ac863d84c148652e895dd3f5ecbfaa2a772b2939b6123e0d2521c0a1cc002087dbb5206cd9f34d22a1272063dfa13a947ec85426f40137f5f82d57e3aed2349c03ac3be3c09e631816bb1f57227a767c3cc109082eb0826dae4ff5cc4a9f6794bdf731ed5b8c6afecec77d37290e6e073a0e4d71157090469358f8751e4079fe23312932d327d4d047f9c323c8aa1a6f34fca6bac2d20d58514a362a838d574c51f75aee6108cf9321dea49e321f2cbaa5918bbef921d7ca7c45e431c6b9cfc93fa0b2cc25374e75dcc9c4618a288b753471d9e868507b204038d0dfe7fcaade22dc8e15dc0ddf2dead53cfe38cfe7dde30d8d8ce540d875e83a9301fcc72a67c965b3fcb74d0fd907d5d522f1cb09bd5a820cdc49e258c3af2c819da3a45e76ef404d5438392b941d45d19891a7e6169eb65b95394065a6c02dd8f4bc8b164788389f536664b0627edb5c5d359d84ae6a8d3d71f55', 'ethKeyName': '[SGX_KEY]', 'secretShare': 'ffc9164ff9e25b01e1c723cd370205a69a81d2904487b0621b0782c89342883bf3483e5a1a1579fd59b5db4903f352609118246b44d57cf6e31fa26e93e3dae2c13116ee17f58732c94bee89a65cce5d8974b9c46bba19eb59ea60d8abe689fd', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 17:20:06,028 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:20:06,029 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:20:06,031 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:20:06.031676 [2023-09-07 17:20:06,040 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:20:06,040 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:20:06,046 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:20:06,050 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694107206.038683, 'te': 1694107206.0504963, 'initial_status': True} ====================================================================================================  [2023-09-07 17:20:06,052 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:20:06.052779 [2023-09-07 17:20:06,058 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:20:06,058 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:20:06,276 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': False, 'status': 0}} [2023-09-07 17:20:06,276 ERROR][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:126 - sChain: free-basic-rosy-SKALE. Fatal error : user 7 hasn't passed verification by user 15 [2023-09-07 17:20:06,277 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 4 [2023-09-07 17:20:06,277 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '23849ce61f50da85dad35e3863c5d0a78a36b80106c5cfc0e13d236084759a310019e165505acd747c77b7567c8cc14afeb44a18e80d130abb25b7e69f08290a2877ab2e04729aaa780043c8734b18b637b894bbc5da59bbdfb8fc5a8d9ed6661686b3f309e7a91913deb91e414174cb5e37ebbaafa62474d2cd7b04c6d3f18d1b9137783d1495500dfc5040b94c2e0ad2f35980df6c45c56e1336ff927e7a3107f91b2c10f5803ac9f411b76b0bf23217c9c36d2f47cf55448c3b83948400de2fc43f16a95157b82de6fcf2b704fbdc0c338c47d969f493c52687cd13d7aa5f0c251c586186780cbdd0c99698c7e6a29cfe967cd55dd234f0aceba06f5e2d05049254f2006fbdf50393c2560abdf67af564d3c31df60bc3306d10190c2ef6582d7af44b069687fff29528e0f8e00278a97eb0bcf90f10616a5099636427044814ec33af9faefde51b165a4c3bbda8d87c0fde92b7871ca78bc3d9590841150126c63c981a24927be53cd722e085771771cae979db2c3cb49302e6b3e733611e0d84d38d1af29282da8f8ef96f7f4577753262126a2ce8512073b9ff6125ecc623f627d566fadb3cdd385316e359b9a9ce068fb9e8c771392ca3580145b1ccc90954e1812abaaca7a4b411765fe375f82e3f7ab4b653c7443dd19859757ce38f2d878952471d8e026103afd2717e974657fbbde61d904b6115921f56800ffcf401b55894887a6b2b6ba610da605b0fc79f226dfa5dd14774ac2b421795f412682ac610ca88ef7bc91dbe91e196a75406e6273b5698595aa793cdc6d49571968d126395c29d8d5bd92d6dc399e032118106356e720af9005889cefe997bebcd10299aa348d9c042050f079c47f84fe7d9c9d4de9d9a63cb8f685f3250d1971844028c82048f1937d71672192b494815796eaacc786ba492137a808c849337757f04ec123a304a71fcc9fddf7e8e06f26e74396b8a5fb61939f7ab88a0b2710cb7206d1e66af2b0661829de0d67fe6049c8bb27b271667cfd9203ab8c2b30fe6a50b204a65db059523fd0a9eba5750bd3d8f086d3a44efc8f233e0dbb1166c55c020669e1c7994367b060eeb41ed031f73dcc5db7ff111376419c9fbf9a49523f31b01944b10e065930c33a67282c4409dde0d9cb072e875e0c9660a799db3808f1f9c9728ed3f6695d585ee131496a84feba66874b4f5be62097aa9d1e8efb7871d2fc3393825446e8d7c9a1dd3e4bf09ae47f272d694ae43af84b71f7068f3132b03ca9c564262fa0ce4dcc1d622cd837ba095ab1cfd534b0fa92fbbaeb0dc41189e41e96682d3bacfbda10070dd374abd11f14d05df739477f90320ae892e431b049d6436cdab979a4810e1f3a37d0c25a51d8c41056510ee6075627cc2edc501ff5521af3f3e6a7f7f2450ea09f961439f56e44fb73bd0332e199b42c070a61477f07058fb450d91a6479728c145457809255357d8095d94fe878f8a98a60d28ced9da3f407dfe56b8ab1e1b0b4db20a52cc4bec6b70dde3719658b2739e972e500459153ef764dfdf3e89b2490178925c7385c22f0c375b1923e6a426295527655636eecad22f5e340f5c92bef80b8cd39884907f1b88e55ad7af415228bc2b9eb51b7c118c4541674f313e026ded6142dc02a35c78251cb66006148dc5420e1323e91427637b5d4786ae863f0a0bf17f6c91932ff05e35fd686c946d704c283a4bfc6ab3fbb87738edcd9dabbaf0b1cbb634d71872a0493e4ce4ba467ab5164c7a43a252744e140d000af14a6a2c3a1a1819cdc5fa3b92a60fc5dd6612272b1c6b6a9485afd487526d77becce92fa609e3d28413fd10c9dfe2a8ab99b46f26e27a2bc6fdda135b2d0304b10d93167f8f2acbf8e44eab514ea231b16bbd2c0d92295fe30d5a0b4be5ecf2bb42ce1298f4483ff097f47ee520deac1d9109ee0881b4fc39523bc8ae5c862859b703dd84d920008b6dd64b694eb688c463903f', 'ethKeyName': '[SGX_KEY]', 'secretShare': '62d7417d37881b5d9c5e03711933ee3970ad981de1127fcac50cc82912a9c82c4654e4afbc7d79f7b01c03cf73eb356d2e54696944ed55fe592af5311e202b42a57bcce9ecfc6b25832d2f093138aa983aa245d8d262913d68610d303418a1a5', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 17:20:06,526 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': False, 'status': 0}} [2023-09-07 17:20:06,527 ERROR][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:126 - sChain: free-basic-rosy-SKALE. Fatal error : user 5 hasn't passed verification by user 15 [2023-09-07 17:20:06,527 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 3 [2023-09-07 17:20:06,527 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '0a0d98a7020bd98ffddf6c2888db22be4da6cafb818cbac0715e05b95be380510757c04f3b739bee263c85cc07010e8fb0daa753da8c6ef3c6bb7e0ddd99cdcb13f8a23f6ff64cd27fe52c06e5e8ad39efb3e66227f3c866bf140779020e58891a49247d0d62d0a664d4de57f2badf558fc2d04079b2ea99a1f50b596f99df7a103450ad5c6bbdb1433b3438fa7d449d55f7149ed4c1865a13575b0b338f98d8237eadeceefcd5065cb3b7e0f4b79e30aa637113577d99c2e3264080e5e73971114a8794aa9b79a086dd9174c12fa4d564e6f71592ef518bcd36372376b0acf31109022d1b8963343d3b32cbd58ce08b7a18628d07cd38059026c83e5a5a1d03295f974e5117587cce337a74a1882418dcc6a05fbdfa5c273a2da5e87df6c9ac0b175a66eb6b5008db96df6dc0501ce3413f90af38d2b8de101ec35138ce5e7d149298973318b459204a3cf48dcabd65f28bd4ed574020e0b1c3f06c62c43af52247d148e62c013eb59d8192fe033adc84cb1d4888ce16fca7938c40bf3aa47c040f46a0967bace8b80d93c91d9bc737dc5dd1c51182e0e2ab2c1770a3507e080bbed0ff5b51c3824d42ef89b517031e4cb7bff5b6ffdfb5d513421420a13a5d1bcb174fec53d844c8e7ec9bcddce5b09a82751a67437580695c9ec662a80a99102571355aae21c3a619f01f0437074d046711b424ce33ac73ab72f6b582e311084e1ae167bc03298ca23a2d299b256b8b7407f8ef8704686add5b7568e07816287563cbad8fd79aac390feb299f4bd03c2074e04bc7b29bfc1221159d8bbdf71f4e479fb651434edfd8faaed696bc34e719c893bd2a5fbd92f0e2699ebbc5452b961d10e992e2545032f5b23a97ac8532c05b278e255cfca39839be2c8d02d411032d5f287717ef9ba830bd326a233eb14867f35baeb745af74664c2307f158130d7bf855773e728828fecbccd0d3251fbcb87f9e8efc4a10454eca26d4084b0d6a4f97a5a68705cbc94adbd692390acafbb8b9b625883dff3abb561d70a0291b22a792e6fa21e67ed3bf3516e525c1ad948c4e99903e730fc722371ac3ae970f86bcb5d1e444b7f472d4c1da8c02449fa766b13faa76bf2de8d7dcf774a1681fe171303def4d8efbfad5139036a38bf04dcb7eff2c5e2b528fbf16a1a93fe1180f50ad6a91d6d4d1c220ffef06e379595d99941248779a4006cf4e15e0d6050c85fd38cba690f18ee305a45dc1eab09ee5963c9e8d62679245bc402418c9a10c0d054516d67374ab6e13de16db64d9dd1746c571ae2989736785e3dd470e3a1ee0695a4d391ac064f00393bb3c496173f892d04a99bc26f312d883d194748d26ed616a5fb5c37892fb55f13cef7799b10443666f217196d569f6418b1951720a2aa6029a27bdc68426587d066a29edab3ec1142ca08ede264629459557b1bc2a2025d6600eaee61f12098f557eab38d230b71f92c3e599ad6ab319a185cabc0ecf8c0a15dc820097462d42b95383656c41c1fbc2b4e3ff47baee2a038d422e14f9fba5afb70d6fdd42bd0f882ba5ed60352283fd03f5ca3147a40d459b931508b640ae3976ab667317afb6d7331be0f143fc43f88dfcd51ebc3583390fae561da265cad4285c2ce0a20199e05f60461f861e63379262fdb1e946404562054516efdd793ae4fef844ace2ffaf4d464f3315b4337f3b154995eb691b9afaa5841a38ed43f1fa29dbc2444f5c0334be8148db5404b09eb3a12975fb6c6828a4a624b511677d2da7649f40ffc241fcb64aa00e3f403370ed9c1ff169c66e428de20b78779ff63ff3d7b9976fd1454e3ca10a8cfc08c56f9377e070a64357246a1b29a966a420dadb9780bd0251b42817e5ea77eac0654d514dfc77c81c6f8f281c01ebb35da96d6fbf1c6d9d3335467b9ab1c5902dc5e766f48905ec185688f9b4180e94bca7366b0320e112d202fa459d300ea74490f970d178b5fc1368341b1e', 'ethKeyName': '[SGX_KEY]', 'secretShare': '4757467cb38ef6c703aa0b12af7aea5f8d2f15fd79055f91302639c7a3c9dc33be1845e0231b4618e50de7b610e43aae5fadd3ca8730bada5c3b7ad6d115911f460eb5a457fefe7dce969357e009afbc48052d5667eb0c2c5abf4664f31b34ca', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 17:20:06,737 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': False, 'status': 0}} [2023-09-07 17:20:06,737 ERROR][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:126 - sChain: free-basic-rosy-SKALE. Fatal error : user 4 hasn't passed verification by user 15 [2023-09-07 17:20:06,738 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 13 [2023-09-07 17:20:06,738 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '03747d5a16d7fb693ddbcc6203fe7a8d38df4ea643058f0a67f22c845d005cb2106947afcf7791f955caea0d0c74cfe617d923cd86976ecd33bfdafb42a994aa09279afa309ef94c7c58630bea385c420ed1142d559167d7f55c9c792f2e680a2115605277f3b86370acdf2267b3e8e7e8f896773fd3a5ae5a19ba6ab1ac79ad0ab38397b984cf9a3c49c00613042a954b84ad425142809bf497532c8c4a869e16c9014610e73b5011ac5024e30896c34edd2c41e02d677dd6fe70d624ac925109304eef7e6b8812f0d72bdff8fb15f50c9281741f3ca230aa7525bcf192f38801aca375a22dbb2caad3a295f1392be0b9ce7007ff4e4545759552454a6474aa30529f062fabb4ec811b8918766b3dabdea7771875c276a89e84db80e9b7930c19b539ea3a9259c551035d96c08b789632cc8fc846627318d55de8259eadab3712f25e0ed9cb44478a3c1a019c27cfcfeac07e7c789befbc7cebae8bbc081d202d1fa086d61e20d572f19dfaaaac6b788aba556324fafb230e304e4aa97ccb7e267be76c1e897e31ed54c74d1478630450eb9ee87aa3263989472eb3c0f04f7814713893098aba2c77c6ddd70285820c1cf9400e8e5d15809ea6f2978ba31ab619a4eb457003ee3125b111f29b7b7e48bb6ca6dd56bbdea04f38aa11e7ab683a2f042aae0fb357a915f9daa379cbaa5d695c0bed124fee35d0b049d69e57e1882dfc1d9b3f822c0b0299ace63b02baacff9728936e6daec56e042f3faf942ea207099e43a3a0ced875ce409816e1278e3d6f6707b68113187d91eef1a562422408a5974c7c899b7bf581c830929dd8001ef3202aaeab05d35b405d04d012be0f289f01470af8f46e8498cbec62b64426793a66d2c9de84528ee0ac71016af2c109e0377a0164bef22d64816aec373db007b6f375dff57d2fb7fbc078dc49de102b4cd5295b81e35db5e9007bcbff721849da543bc1a6b8940be6797289177c8e1fa5e39d1da538147dd30ac3ebf0c420049517065a13957fbdc7ef9468214840062e8729ff91af0018d09dee1794aa7c0e46cf785dce6755f1338e090d8d92080c08910aabcc48627f6c4d10b25121e002cac6d96033b25dbe20b9a68de67c4f169724cfe79eae5951e5c7ad7bf90cc08c17d5a1d53713cb2a7b869ec785fc0d0d38918bf127b162202693a3f6bf3c52462e8b602f407b9c8f971d37b9068a7a10989cf5540b5ffbb94e89964a053740514fcd5e35319d50944ab82a9b1123100d28fc2416877c1d44e1acc09852dae83669ce8738217ac65f8ff44ea15eedce25d4be530d706aae47b3dbf136c7dd3558f51c4b0af4cbb2c50cc5f665233b76150c1af19b2e23bd10f55f41d3bb722480a52194a46796cf665e0eb635a30e8f0af9f85257f0e7f1f73ec77212df30512a7bd0f55b5d3172ea97eb3712861e4b212d95ad78986af10fbf440eb40f5a4d35ec53206b44fb98cd9efa7da8a606d62e1992cd3c3784175fa0ddf7cbda127ee7959b10b6ee4bbbe5663ea9e389b9dd2b32092a882c959e3f123330fb5633cf0bd37805650964361f8ae313da7cd73f19527334992437168d47518f3ff83c1c69ecff5878af2aa887737afaf5b0b49f0e7e6809db314201243cb872d3a4db7b785b5e6b1a30977e712ccb28b0aeb8cd051d98117367ff034438df2e84386d66daf328643ce2f9d6543c126260dc752c1beddb46c539c9aa6b28c18242c1077d4de6785cd0b439b5f0e1adc0c754d0661fdbc2f06d858fffe7889fd59d0a8d3f0d04f265c17052248307aba412f11b4b0c9326ce83302793c5227184fe664596c9cc4b1ee0ea0d2b0ef76c0b589f2c4a23f070e72631a53536f24cfa419e93bf97be68b5436aa41336ea0dbc94d0438b053e5afdd2975569bf9833ac933a54df4f67ccd460e7455c5b0378bdaea5aa621f79a553bc83e55743691e6ac79cdf481ec6f292f84b79eae96028a3d04d739d', 'ethKeyName': '[SGX_KEY]', 'secretShare': 'f9908b6d2fbda9d2429602633f42f8d279a0ddf4b94037cbc35ac43b7a218d093504a2df9b3278e5a7a44795e7bb2d560294e0a793ed06d32299001a0055203a6765fb5dc627b8ffb543896a7e5581d9c272da17e693de0b7553ff3c41db5209', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 17:20:06,938 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': False, 'status': 0}} [2023-09-07 17:20:06,939 ERROR][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:126 - sChain: free-basic-rosy-SKALE. Fatal error : user 14 hasn't passed verification by user 15 [2023-09-07 17:20:06,941 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 12 [2023-09-07 17:20:06,943 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '0b3101bb32c3d8ba4adb1fa987a6be6e3c4e9eb4d34f375198fcc17b0b3addff276384fa38c4cb4886e0f4c09349657ea1d098e9d801f14ac6981c4ecc45e36d13afb54a3f3981337748d3ced08a7860b99e9bce0b26443dff55ae539994cba52266e3402c957790060a9117fa7f10abc56ebb586f30e0f58b2eebb1bca84c861215338b64ee2472d5de4e26f30efd4cf69d1d69eecff8c16285d9649bec968a20e7419dad6f820438fe8b31487e2e1b09bb34eee6759e71e62623113753d4422cb1e806eb96d33ec0d12aad4bf46f2e9b1f266d526444e0ad443a51b7876e5f145aec760e731613a8b69319ebf4b4725c0178b8339b3a744c0ced00c93f4d232ba447a8f70e6645efddc3d6eb5c83ea415ff2d49cfd4dc8c3d2addee6898b152e3c13680f23f41f7aec096262a8cfbf90352690589a3b67325adf94d6d418b3220d3e7b84181c80eec9b29f4ac5b67f44c789986159cb6edfe0701db33fbc6a0bd9f822becca32a0492fea42d0f8c742084dde2d72004c2b619e3cb8a8a6fe5114d9e813cc0506f61f9dcddb9cf7c51aee90ed114d518e4e6800ffe536dbfa50462dcbc2279ff8881c4acab6c96abcc39d62f8b4c05f4ea34f7757227a45b4c220235b7f9bad1fb877cf1b21189294c7d5e2ca5f34b6dd4f3f4d85010ad1f9923669ea0626d9672021849dcda34041d7527d83f309640385a295568e766ae4808c70fea4bc35409366da98d4e3d1fb2db27b3a5433b0665ed909ce12bd53a2810556778f3374b8a7a4faa6799fee7cf36803fce2eea897b6c51504ed2ff6fd322228293531807c3fe2f376124bcb7f51ff15ba2cc56e235218abb4d784a3f862856a93646f534b5149acb06f3a766cdaee601bd6db8e92939de0360471baecd00c143474f4dbe761a190f35432796518b4e5f633eb4c16663b9012d45da659f1bb747646187a2ee9d2d2a1edd9bf797e0e232d3038847f1bf943bfc700e570818720ac6ed991581c729325502e4303f064110cd80a1bbe460e604782732c42119c1b3f7c14659eee8205f1c849dfb84ef40f674e31382a4e70693ef57e18338235427e7a55c29b098285ffdb9e3a0366f673a347a8a6288793e544364db4b842a850eb291e6c9030347b9ee8a21e746276bdcb0042d0235123556eb279d14bf0cac7ca5decc3493648bde3af25760266881285a733b62513fa1274ce8dee1de233502c4af6de80017b88c935a57067c98004baa6dcb0e6eadc9dd55048c3f5c1ebaf91bb3f6e2ab49faa1e0847b2c0c5ebbc571ee1a151b4e3a5f1d825aed950a5110a269c0a3b4701c16a53dadd8bc21985992e588ac97d68c132e4b1053c22595c44677d1fb6b5a8d458b288f892a074f12d1de983d6d1889f36d15eacba12187cbaa1dd997ecb0f21503482918164feb98f8f23c89b85ef8cca09de8ca030b18e8a100dcaadd752d0808c42417f14517d5d5949080119baddf3f71bb45a220727f0193c7a7e7becff6d812eecd7e1af569e5ba705957a8490499ca857c7c0d6a63193e1bc41ba64efc77f6760185fe7d72cf496dac80b21c82f40a00b5240b1212905a9e73eeca244457b2492a7bfce5923dfd8f688d40baa4ecd48673281710ff773d7ce94c90b918a208f159eb9d28866b3882d42c00888dc5ae9f8ba01df2249cafb41d46f41fc6189681b928d3d3fb0241973905df4026f2475a760414d5ece1311a5589f14fa0fe1d26e2011ee922225b8ab4db9ce86c7da1387c2e266d702c3e3406212617a46ff1af3b7649e16a8d838a472ef81825108383e2d40987abc671eac240354eef8bda9cc352ece16ea1dc1394d9fd0d566f199eb1f51046d89eb4befff5a486467c30bdfe3b3d59d9dfe4898e9767e8263f715c49b5106c04fd167d8278b4d3a25c78fffbfb4fa1ddf71b3a7709192c63c3ec42e00a24ab17323e5dc0c2ccb035aff6e71d726b817de8c144efabb9f3004bff888d1e', 'ethKeyName': '[SGX_KEY]', 'secretShare': '2509437fc5b24c0c318c092d63ebe979ca0f558e8fa6e3bd9e330fa01d9d8b5152e7e1d2a2f03a85a5b215bb26571c64e82ec89405ed684f399afed4aef28651d78d2777fc9ff302faff3e3722dc31e418432d5e240e40efce1b0ad779a60b19', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 17:20:07,171 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 17:20:07,172 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 12 was received and verified [2023-09-07 17:20:07,173 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 12 [2023-09-07 17:20:07,173 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 7 [2023-09-07 17:20:07,174 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '0f8d43d1d84c3df007f50cd6e91ac6c28d1f2f87acec2fd0dc492910d82a996e2af5de2fc36c9add0976474d2f8684e541e59c7f1296fe5fe781ed8984015021210936444c7e4aa6a2e17e06da36f993d50665c2291612e24ed6aadb60ba49bb208c74af55a780dd4d5966d1e1b6c86c463861edbd089a941113dd1517c3789c1e2c69674bfe564d5cd7e36dd955910d889ce7d15eb1add61ac1a0eaebcdba7100476da9188a5fb80d23d387e10d95f708a88703f8ecc878aae7e3e80a12645b07534ce3c3cd645f5cdc1f927b79e176a44f5e9089d4f356bd3b50fa6522e02d1cfb3cf0a82738c06980ea53ef195de270c2e97a73343615950575ad6333147b1ed50dd2933e5db6b8def774a9a7610593eb0e087bd6f77c7fa31576c61ad5e81838c9a40f90027caecbca2af622f88423c808f443f7c5386f582379c98111af25b3d6125616daeba60a1647b59c5fc8203b0b6a82949abdf524b0fc8430486b2422e92f3b99b5ddedde81ac6fb649d1177ee445f6a5458ccd5e98270f512268219a3fc28e407abf50e3863551fcf5daedd0a0529d217993bae4613d51a0704306eb912f942d10bbdaedefaf97f85a037c64945e875420b4e09febb6c26fd52c150e462a2666445c2700dda32495652dab07dc6b1e677f27c12ba1dc2c04b5a31627d2908a12b102e47f49832dac6afc3cabf7fdd4654bf218215963a403392c1944feea02f0b7f95b2e028d508fe89f9ca882cf74047eb4bf27f404940c34cd2106fb0fbcc986affc092f3d8790c7c15bd33c80df4d09da36b2ee83e9d60dd5220073064e2903214b0fcaf28867d6370a1119f6c19ce69ccb8ea7cf4e9be3150ed77bb4d6097fb34cdd446cd3bcce972288c62a51779f96a9c0a5da219f047719e5152a6aaa4fca6da3d86daeaead00d4f2bc275447b8613cd235186856f7f829c8c0fbd87171349c66355b3caf45cec1f23468407c27d021d647ef097dcc0429bfdf6c9a76e835f9bf23979cec99b8a56b8e1e612c426caac799f0553301f923f59ffdd65a407ab856a7ab79a0c1ac7450422573f909ef318e975966b22e1a074f2ef9e73359440e05477a9790bc9e0be3c75e39c26cf44bef783a28ace7d61bf74435ec0f39b846b0d736c0a5f193f1f32bab28256ad1cd176ffb5be68f571711aa66acba20696988c76bcfd5ebcbb23e35bef0c88f1b194b1b357ea9ca200e288bf44aeabf740ab28eaf006ae9e3476d5161ae7e26d85903ee53287e521b11d75e713b9b473dda489cba455cf73d3c99d0ad2e833bd1f2fe233085ddeadb0b3b80426ea9614bc363637dd00f391d89d33ddc07f2fee33d40bf3d550f5ae7101fe1dea4daff30baf2d3744053e6b96ae3b127f235a061042cdd6399fd5c741e9a610f1fe4fa4c7afa96725d7ff0506565a3f2e137867d758cd1d36b232ba212ab4384b0bb4edae0b6fc7a6a32b4e169e63207ab5c65c5576f1ad9c6de10d8188e8c1058e61000d7eeb4d7dabcbd7c38d4b0413d2b5aaff587ac0fae57b64601ba36dc29d03aade7a63873fd0904dc23f95d1f15b64bbfa11e32079b00ee560b0806dcc0eb4e1b4b6f35c84c4d1cfed0191eafca24c0725ecb04f1f30dac6126847117d3490adb24abf3d279af316f3f70a955ce1951fa325c0dc671539faf1ce2e4e6b2ff52396064584d30c94697ba8aee07e5cfd877a5ed40af8da682c41ada34dedb927573136727c8de0a07f2ea2db12de9b31fd6443371736811529c1dedbc340fd94e03c6da682eaea10ba2a3b1172ee65b47f9acc21b3c9bb5f8cb1e0a8abe9d3971c4a5219075e6bf940cefc664147cc6565f25ac67a854b8bdd61392a4384c52d681e1219bc2e88f8b63acea98fa035d80c6c16468dbfa7017262cac497f2addbb8e8359a43feb5b3d07cdbc98855f93def4d4cde48145354fc2078d3e788a627f446b0b55378bb9788a0a3d77ceeb30de36f1f39a5bb11a5cfc', 'ethKeyName': '[SGX_KEY]', 'secretShare': '262ecc586e70fa6d4d3934838f4b76cbd9f09f31f70cd0ea684c29b2b647a49e56db3ca31c25aba39ae6a48faf2c2e1bd6337f9432e0dddd56cc9a0761a0b9d91491e304e5413d71cfb64d326a869d7b14b2780a257f789d7069cf04bf7391ff', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 17:20:07,406 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 17:20:07,407 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 7 was received and verified [2023-09-07 17:20:07,409 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 7 [2023-09-07 17:20:07,409 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 0 [2023-09-07 17:20:07,411 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '25ce2aa779544e65a247f946b92360bf267b16631d175112d9126da6e5cf80ee28003746c247bc01cc7a7eb6b08660e3d7cbb518399b1a2d3f05ce99fe82547917530866c65c6e262ba8ee39b6fa6a52f948cd1f4d8102f0d58d96f1313bb319147fc2ac477c0a3828bcc2b5049e2a01e7152610e759ea707219428b110952e11d9bf4fd774b7cd0e3b4c38693f23b54a936b217686feea866fda9b342e3f30417268dfe5b748de1a2cb4ca77459e167b5244a7076a98a4e70c4f4ce7d31314702dbb189cdc0ba1c0520cad58ff928cbb1eb1279a6a1f1b4c501d6abea8c5cf415cdaa23cbea26c824f72735612908a6e2eb4466ee7cb7899c22a6bbc776b85b05d43fda70af1f175d307582e6fe2dd25320c34bd4bfc809b56b63369672d6311a1db9b75b5ce989017e1359ed40cf8bc18ff3d6e077c717369e00b94f20199d2cb14e7f49bd628d93311f79754e9d3febbc9b5b29244c34d8851667484a8043259ad7b5b78aab1887d39adb5c4ebd0388476dbe530ff795e3946107f98ae9463037bbc8f4e951ebd6962ab37e3c8f0769d15d7b06cd382a5b18a0e9fd3d6fdc1fd6b0743da2cb1be9805819c32a5c34f9744c7c7bfe2ea9db9970c38ec702f528e39a94c164de4d71db1e2a8ee111bf0862f63a4860789d98b1c7f69e99a84407dc6b1acc12788604ef3a15f9f42b09114b5438bc40336a2ff91b99afcdd9a30c173070cadee87aa292adc43c68910c9c4c8f98afb3f36a08b9a1ce27eb8dc0127b1c9c88c61477ba23fbd55e402bd80ea2944918480edfd81943e0e17ffd5c2cd6d952533df51d009f7a4af183e915b697579fc44e8f654383ba7b0386b84d2d05fe4db48468b487b6e260cdb12af837f8f745f9eb8b52b06d77d9686bc32719c8a5cc8acf8ebbc460a53c322b108e3f6a5b3b20c3b922a0f34e13415ada8d1c48a5aebcccdc7700045444fa0ad094d5bb54279e5805e42897cb5a475fd18c1cc009371b350e6226064f428859d455a69e6fe7b7af29a2485ea0ae196a7ee928b3c420d605721e950931b78dca48488f20f13894b0b721e4d63f5c7c95244c10e69c2d99d31d788e3f3ea546544fbfabdd0a2c02571aab23bc403ec54117891f032a7a5fc4a44c658958a85af6efe1d0b03120bf8a893719d923da097a06f62e8850ecb2628d549b1de6a10b74166a16ea8ebdf977465aadb0a9f75eb962142f1ec358a5942aa241a72522df6eff9984ab2c543a787a288970ce738b3f312f0f05dbc76e50769c19d9901289c7d62e726c20a1d98c710b8eedfdf61bad670d1a32e3c820b03176a1ceb3016c418fca1e7d28764b68610165fe0bcecfa00f9f1278332058260b01022e77459c4c4adde51f8693b237bee47d6850b15f36d3fb031ed6aab1236163de6677c633c9134b1ccc36b2cb404435d124ff0c17ec83ff013fa1b4f4f44691d3e7699bf70a6db9e6532813b9690a73eb13ad8123ac35382db706936ea756a2353057fea72142edb271a01051f56aba2ebbddb1cd9d53a50bb98e9ece6f9e77c68ca884251947d708315860cabb9f398937aa0fbdd4488c1f5c3331436b67fd9e147f7828f0e654a29a9be50dfb1df2a0ad2d194fb1c38628e132aeb2efc3a1bc4aff388ab7f78e3d846c563ba71311efda6f6a6b8bb8da0b76cd0bfdacf32f787e502854369fbbc820d20fc8ce414d2b1b0dbc631b094e23d94701504e4c435f09056132f9ab56d78d95bdafcf7cb53564665a03e2b8071995cc3e27438c906b43fe03a447dce2ea641c5e307c00ac769c6ec6f60436170bde535353c919e6e2561fd161d73734eb3f0131e146dbc2b7575ea5a701f1792b53b14c3da152117058d6524dbc926e0eeac654c0635f1381a901500f2d1dcd21ea2fef50ee24e07491e12b5492f0391ddb1183a2498a6db04bb9fe55bef32d14d3b729747865e29e82fd4bf7612ca256aed871534da3f6d6ad2369fdf40d20', 'ethKeyName': '[SGX_KEY]', 'secretShare': 'a4d949fde360a45b676e5084d510f81d96a6dfcf058643949a47af6fa2e1f205ca5b2a8ba53b97b5976e8dd3e5784351d866b555e6f9e2e78699f4b1615c1babbadbb214939112586cc317756270d9678536326d9abe127802a4663ab354b9c2', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 17:20:07,621 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 17:20:07,621 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 0 was received and verified [2023-09-07 17:20:07,622 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 0 [2023-09-07 17:20:07,622 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 1 [2023-09-07 17:20:07,622 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '2c132c7c41b8a7e1021725844787f4146db718b1ca95f91ecb96dc0335fa88f80af13505c0cadde70d05129889d038b79a16c939b49c30d0cc29ddef86fd10ae0e457858a118919e74182c36e83e8059eba144c99cc7221e9715d0f7904ab2bb1b2e8bbf992d036fb952e0d34b7379953f094808900c33ebbc8559e80167fc9509ef02113432d351068176e8092c79c715ddd6b72b4d27f41b62ddbe3e20869d259e850401c2e2943baa5eceaeabe31513c7cd911f8d9ce2f04d839408fa111227c65eb63ba418374efb9f0dfc6f3e8137f7ea3623fc3d58b14d86b6c366d46321434cda57873db5f8a6f9d7770ced141d797e375b43eb1fd03c5d18c4a52c4018df553b6a612e0657c2c0ee7d5248de579674e1cea8ced13311ca813a1dd3710ca0b17a304017857c7b04606232d04077fb993973e15a0a353d58a2bdb0a4bd0aa3c502493bb80892e2422ef55fb4a61ef9d3a301aa30fbaf04ac56954b8e0b2907db7718fabac8542f8616465d5e5402e49f3b1aa8f730e140e3a37ea0f993260036a301426cc91cb7a3d0e46069afa7d0a0039f809c1de40ba21ef43507bb07e7945fbb17663d7c0c093bbd16af91cc8d8c9d9a8affced7905856ad17a77c02a8b9fad32501135cbdfbe82db4cbaf26f4b4bf7c11738d3a5a79b355270685127797958a3e0f78b354628afaf5689c9ea55e0f0d63b876e693fc9b58f7541d185c99cf433994c8ddf7000ba0f14f28cb442e67039b2de484d79c13bec6f3ab1511c6ec75ccb327e89d6c8d866f47584537ed1dcafff71e2f31e9be0e7792d617e2c8611e1d4beb426a91b016e9f953db2f54de194cef3511d4f15cf05b283702ec969fcd8916663132e6d888cf455a63460e662fc1cf1557ccc125d770edc82f5fb935cd1825dbd1412435915b379961580c780cbc573179e1f7ad4c1ceb94147814cb09f11cd83306488395c0fac687947aad8e05a10173c2aa8d009a97e1063d11b0cfdd926f5984c52a87bcd01423977890b873db3a9a906f575f94420c2f8a333c69419ecd1da13a9874057e52717a206efd0bc584e0a613832f2e3b212ca8d86b96df58a51f6bd82342c4fcb22db64bcb9726cd7a52907cc74558f69d071a226e8772782c6fe415c44911f7f3fd07a8fde2a3e52dbea0f175c00bf33014b0af6dc716503cbb09ed49c3cbc4d65825bab563fa314ba06e197c39eb182a2036352afa9f6687f1071a99181f15481d897414272b53afcae0932c10f7780213997e9b3f4671e2c76c442b5f715b5fedb6f51a89b48f644fc399df0a347d9b14a45f7ec06f9cade3fff0afaac484a1b4eade59db100a5c35d795f260609a0f0d9a4955e74f765fb92ccf61d0e9ad00a9dde22ae9a501b27d849f6ca5277bd42d0e38403fb6148fe447edc95260761531bf7178c00a7779468650c3426f28a32c01cb7530b7a562c64096c1c460ff4496babbf3a681068cccd0eefa63a98453111ec7773fc4cd2797e531b69bb66dd00fdff933640d75ffaa1e9f9cdd4e57c90f8c07190321c4b3d59b44c375ad17afca49ed6b599ba9c82fabba9304dd8fe51edadca8c0eb54300a8a7aaf9159806cfa7580690d9f063b4e7af296960f6dc701e93f78664f2b90dbe8d2e75f1a1a6f5f40e48852966f91dd40429acb2691521fef877596371891735455ec6aab398f86fc8227328b0611a8e4ef3e42503aff15cc66949758a0cd3d54c9bad31d61836d02cd355fc08843abdcb6135d3fe39e1c8b974ac90a9d41794ab359e07522c2b1197edd4a7f4f2db66a302b4e85039a05fb9919f6e74e634223446be069aa13a3d8477a815cac9aa3fe3a6ea0e5052d1cf90cc37ad94ff4a3fdbe897f43e683ecfa5c979869675cd8f6905135d48c5d0545763933c96f86e75fa4214ad749f0a1903f2d25c2d6ddc14fd1789cd1e36a0590dac4d3fd67351b9ae7ad34b02913727aa59d5bf5b6c9b9a792154b562dc8', 'ethKeyName': '[SGX_KEY]', 'secretShare': '950df351318d825b0097506b518e42a59a8fc7399f2153106cb7bfcef7770db6df0375ed873229a1b194403964a3cee638de74c8a69b752a2c6b831c2e78ff3aee478f548fa75833742c37a19b0a8b23fd3f8c1d53e15a5f9822990a1c43a0c5', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 17:20:07,864 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 17:20:07,864 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 1 was received and verified [2023-09-07 17:20:07,865 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 1 [2023-09-07 17:20:07,865 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 5 [2023-09-07 17:20:07,865 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '29a242205c0d798423c738d7a098b7c26413984a866244bda9661e2a9aa83b811a0cf3954c5eafa7e0dc79cdb1cd86192e654094c246a766649275cb9e2f72e5119309cbf5b774f0edd1b926b00c81bbd067d0198ca4ae206c23fc16cd9cf00a2d049145a1464853f8395a65a72eaaa48f153725fc417874556d1120dc28189a0610aef5a549488886b3525b26fe75d3d14534ff80631298a930ade1703e907316d4620250a16f0460e70a9684b202d1fb2336b57c9e83f798d04fbd4d158aea2433acaca65594ccdaebe8afbaef8d2e548881d32892d85257694bb98bb1f28908204936bc27e7122a1fa57e94d78f8af6227536a7b025a45a54b61d283114c6271b6445748961f326ccc83ce9a57db204ab24ebbbdbabc1033d43d35a21a991078f08ca76370ed04b423caaa216ae67cd80e4ec7c9a3a643a84bc09a4cef58c0335c8cbf08f6ec716acc328ec6be089295c977fac3f7f25852b1c88a4588f80244f7f2982f15c47e8561ff44adc759a99b425a095ecc98c916d8fc38552140f1e5b455467117ef2eeb4a24a778a7a193cc328f8df69fa2dd95d1b13c9f336ef08009993c8394e7a1f6170e416558a58994c471b1951d2f5033826390ff81e91080bb1feba360865eac63bdf93b49e79cb13f853ac971e612f9c3381e1cf50692c261dabe04b77e3c3b7db5c236978b913cf46e3eb44b37f0dcecf5f0105ac34198760b907dbbdad668d4a2eeea4229a385c914147f283e99aa7163045ca62dd1baad22326388b84a5986bec46862eb5ce4271fc958f75e62bb61093fd4e5666109d724a881aba388d6f0f482da6ad1f3fef289b71599f4f0a2a7a39c0ca1be411d8cb683ca249808d5edc008f05581043fa13371698c4656768af6415532ce0248f34a2c2ad045eceac34b39a9150bb0e6b78626dec74a174547af7f08171b726b25fddcdabe951d9f4db9fdfdc8870df9ee4ec57985d7a4d71f615662138492c5d275a44424a08d45a83a9911717fc48348ccd3875edf3e030ce517e1fcc032a3ad0c298da865e6cd323fed3c8bc1089b7184a5c4a76baa512e8c9feb81e25305769114e48e8115a63eb4c1854030e663595fd2262d08e9f17102ff8ad3a3c23a9127532629099b12aa460c83f77dfefb1cf870483db4e84769f94e8ec5ef917f61cc870781d90f338dbec93ed2f15d78c01a8d7cdb97208e2db21b2b5d47b13a9dbdabb8bc1776fd5996b3dc44a781fcd6666d0fac8dba872be0b304bef0e08218add25c12e2f4770766e19bac18876ad5bc3ac478b5b3f6aafc8091d8ffa07631ce01f847da6f0a16cf2aaed5b1fde1ea38440814a50dcfdb8aff55fb82200605a98ab7c7cea586dd3fff5f122fa9df8072f8c37d5c1e6ae50be6c627f6204a3e9ffedd9bbb4c87db52b1dc0aea389899008454eb2885e57afe35d307e321a5338c8f5262ebf125331df4450fb048e576f3f1a8b5c2df51f38e93bcbb013274387052828d1dec033790e04f7ef13b013c3879fcbfa1711a63ff79c3b24b015d293501d6a023feb8bdfd1ddfbb9ca458a49c25ba4ce377ec5ee1f89bb26100cc56ea583749088d6417d0082802def24d7e24c64efec869753fd78f56ea8f105bd80cf78f6dda6b0308df625b08166812d89982f1c68254f81e231c892671019f75ca19d00d377eaace52139c17cf6c33aceaebe43aa383bd9cc0d7f1f4d28188a1bd90b4f9fb79a5451d9b0fd69576b7b9848275584e5a8d0f7023dcd17441631766bf5a66270d8b399e096937a3c6b7a45fa76684b3bb01472c7cf011b3f009a6dc522b0171218e52e0a21b1654d272222b039c791f4f03b565b14a1a04f1d0a2aee6a27139de3a00ca177ff72aefdbe80c826e33fe33f1208ef7fcd954d1395a7eca1820f8bcf75c1753183868b23682ef31714e316e42df1ec324118282db8e453bb00506868860d98292b5b82f12caa2c91b2da0603fd7a4c82ea215a', 'ethKeyName': '[SGX_KEY]', 'secretShare': '9a1a5836c249f3b2cad18aca1c30e26da5bcacc7c62e16b79d9aa2c32906fa7b23a38dd225946dbf65de17b2929715db60bd1132999bad767a83c3b900df5319dffe39c88f4aa028bb600ac085a4142ebcb9096b0bbef6fceedf56063e443186', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 17:20:08,088 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 17:20:08,089 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 5 was received and verified [2023-09-07 17:20:08,089 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 5 [2023-09-07 17:20:08,089 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 2 [2023-09-07 17:20:08,090 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '1477ded76dacec883bdbe06f82c5b0ed6f8168a2a9c3807becce10783147dcdc2e6fbbd54047e5b5e32b159f204c7996cb816cf773bc2ddf65e753a606076a222a66f563e00be3c823845de01b404250c58d59fa128755e55aabf300f9bfa0a1181248aafc38c55a3e9fa3141166edf0efdc59ebfa0d25e516baf1fa3e19ce8e255bb6235e767451b44b7f4d5521b789723c8865d777f617423e7eac5a94c2242b86e9b0b86c32c7f760614ebf43622b077e6522d8f455ffa4e59483ed2e25b306f182b8e30b5e90408a45902bc524a4ea2e391e36382a6d55fa53fc619c473e051f5208cfac9fcef7511eb42c231d13ca2290e37bdacc8b6a4921b45a233ed626c829fb4b3a953928a449d420893ae080067b253fe6f02acf0d0440346df884256fe3e6c15ab2b2a34c5c05d3a3898b78ea5bc0dac4c1b408d5fb7c57d26b4a2a7ffa7ca4406754a7967d32803d19f6ac9969ee33a9b33f6cdb80461e990de41748768f805bb702d4774d1c6ded6e1e929d742ad67911653822b0e7d83a8f630e3bf1add29fba8b2350ae4f98552a78e740758f24a19657f6aba69b29d5a8c91cca40627448fad7e9229abbf1789e19f50c4c31e1284ce585378ff40d0ae83d10616bb14d93a2185e0c04c81ce5e6c51a82062c50ee19e72c23cb4cb8f182d2097bbf758183f31de0ee90853c411f61ba705c7236a2d616d87db6d30e6805241398711198f28b9e6223b2285bc75582e9707cbd7a97d3c111ec19440035a81b29e84d837d8a8df3b31145f759583d1b9cc7fe170296b5a8d0d90e1cd4e5361c161ec50fd3addcf8e55264cc5e1a5427559235b0ffafea866a87a62628d69eee0991fda55a0cea6250197c4c2443eaa85a39d8974ff76dc00b83f3764f704ba414ced5a688b94274a1ca1905c00ed31aa9f95673707ef479b5c0f68845e6799807141466dab1a23ee3db50ce861dd9cd0807b62ae4cdf16ee1cb000da7ef519324f5e994e565b048934b8557628c10de4c436737939cffc9c6d82680ce473bdf2602c82cca7398c78c7aab82a8eac7019aef69a20c6e1aca520b5261c90fa4f60f1815c9994c8e31c7085386685d2691199a8aa74ee42e015ca67912f06701fe14fc36449158ad741681758dc351768b9906b6726c1c098033dccfa39f93bcf71cec3254323461287925efcd4865995d699a86514c1196b897ba08b3042f221a2c6ee9e278cf2d540e21dfe0e7bdc28101888aa5705981acab4ca49fd1fa2f7718e35378b3e5aac77c09de2db011924e8720a88cbc4075dcd75b5d792d729b0a1581c8caa5137eef122a6cd4722df0e145d1845ba092b5f5d3cdc05febbfcda72b8ef16854df2850f6fbf0c59de00ffeca4931affd27afc6e37b21f59e65bc9a097070e74643d9fa7e9337acc60d6cd866cd7d61f41cf9a90ea503fae1e39ab2084c1369bc70fae3403b2558d0773095886faa8055eef1d06d76d488b0c9ed5c2f12e37a8000f9553a326dfc3ea6f84a79904d84eb6cac2c87bd27ca40153cd705049ed368cd97bd846cca542f7c78f17337fa80cba5dd548bbe62e11ec87a7424c357af7ad07f70bec733b11e44f3cff08744dcfb4f260dba61a950411c09822ac78a6d4f8de3bdf9c8c21d99aebd9b51a6ba3f3269dd1525b8578bbb3a6b6c22e647cbc625ca0e0b087f845ee55cb5a772894ae7c470fb8399ba34e3ab6d520af1302a99d787091e6aa5591773f4f3e75e8cb7bb48066b7303188ed98b9e4a0e2452977d952397bcac29a13c6ec065cd0e91bea88ad8e48c592d97f83fdb12182938c75e3619209be4a19bfffc3a5230ecd607e76b572d908cb8b764c89b0728b3d3ebdca4812478c757c08c2f17bf0a6e3638e44da4bb25688c1326c987b42d507a65879714038014a8a1b3c7637ff96b753beac6f31815390d402c617fc92f21698da86c88f44ec24fd89084afcdac395e285e9a1d2b6572c5cc5a13f440', 'ethKeyName': '[SGX_KEY]', 'secretShare': '334be07a9ddb9270ff81cdd2b468fcff4ab3fe5f2c8c7857088276e3ce3a0edeb74f69ca90ae9b2f81d43df25cb918045189c4795f9e4df4e3957b0f431b8be3eb03b58d9a2462bf242f4fceab2620c024bd84da7e7b8e2ae83775e7d7d0fc43', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 17:20:08,316 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 17:20:08,316 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 2 was received and verified [2023-09-07 17:20:08,317 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 2 [2023-09-07 17:20:08,317 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 11 [2023-09-07 17:20:08,317 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '248da97a225298046fa3c91189dcc7b3e10f9a2d7280a9a06362075449a6bcc818e5db100365373596a140ac13b738cc1429d794899c0266a42dac79e5fdd4d4228dca162a34e002c797bd810e025bcb781bbb0b88fc02ffe286223c92c10e540b55499e52e39f6326dc810920fecb9cba405b7cb884c9557af85408ca5facdf0cf69082a00910ada371033bd1b94545622b986bd21555a38c6077f9afd2117a040881994e9fce300305e5ba67b97608ac54773100bfc26c17447dc9de74c0490e4881db3512ef7223b47aa4873868be1e9e5b721bba7265a01d8175fcc59af81cdc75eb008650a8cb56898c99e6c872b8d8d3622d455d9172bb7fc40ff7b92e2b1dd32493a177e54525e05ca7f3954390f56f9fde01789afbc8c67b8354003c2d5c871046b9e9cb6e5823cf637f77c45625883df30174a6c97629bab0b4fcff0b5f399239866bc9f721cce690a9fcb8e131aeac5a562f006b633b93adad017802ebcd2d7147b43748691c6d5fc53d5232f609253bfabcb677a4d2f1704581591ef7c6e44c83d882a59e60ff427026ebc19e41c5f598ed136ad502623bd917491bcad3c456ad94a441c784dd16e847dddbc97a8f96f0e2200db872482ab138cc2f2097901f0e012cbbd6a59ffb5b686548dd6e101b2e0976f4b2b3923f7f6825114adb18acec73690849e9d5b4023aa350d1ab29278b47b79cfc05c770c5309323a0e42df4d457e14167841246387c2da8302e76301b1d87a925bc1fb1ae75980edec4ed8267c76ea065c6b174358562ab8f837fffd54920a9801b6273d38cc924270158877c2db9ac60f3ccf4ed285bac4b9c6e77b52109180421119b463f1b2f3a8ff8f19dd63e4ba9aea5c6c6fb289c84d301066f23ad8b878184a4cf8ee00ec73cb9c48b70144fccd28adaa9f276ba61ff9b1b209cef1717ee4e44a74d8c229d42b9d0edff9920dc7ccb8dea38f455aeba4d3f504867fbc2631a49afaf050719e4629db4675394edc6bf1f2a26076e3d0e74ca3ef1b07140c5ad49efbdb90ba35778397265818d7b5ccdff8221aafd5c609ee31ec4d802a9245621f4ad2a0f10e618a2e49896850efb9c6ca8df0260effc8f6d84ee18bd16c1a6ab39811813f1329446f650d86906039b2ff4816a115a9d3e22532ebc4e13970a9f1534cd0675922edca0eab818c97f178d89e3158eb0593ab012b8478fc0cd2c3f8a127828c7fbb3c6c16dc8465671b44f1903b2b00b8e152a8f79ce3500a08175e6c1ea083b437d79da1081d046f3c8f0075f916b4e38826980671ad433bfa22b1d00e71e9cc07c7f5eb817998c5755327c44979ac9e31d777773e0f69ba4fdc4c2aca32259ebf23c95108edaa6c576afb430f4a94e2038c23ec34e13b30581e7d8827318b0d7abe3d42d0e74de1bd825959a0aa4d7896209dbe1e8ec843245a04566232ccabd583b9917780ce442597bbeeb5c0aa205140062f27f7728d8ec037eb37209f42085f2d98c410354f5f8ca4ce4dd275d20dbf2a65acd72706e134ffc035500c667865ecca34fe410b04d75e500e091c419e986a2557bfb1b193b8e149ff219392e85f9235d854175395224fab85e805873160f11ee1293f394d323ee88170e0ad4991c07847e77e129f8c1a298c4897f69c5e97a3d0d76980643edd5930d11d21b10296e36e45a57119905e0316183339209d0c1ef02549488980195f53506ade83e027d047e91841b369d3be3bc5f6d2cca9536d93523bddb3011a38a661b8fadac7aebe4a2f1e44d2c0db4a0cb85cc2e0fb55d09a0c407ee329c6e08d11d38e6d8e9bd52abcbf0bf3905550c4be317621ecfb70d133dbc7da9af61b7ef23f728ab126ea9c7ec1f833c19ee0c87c5e27dc3acb2480386bd57cf656c00a40c5401ac834847a87d19f7ae26935c49935dbf652ca314f93453e8871c03c1ca184658a15b82e3437fc101d4b336774071d589d6143ef2f597e98427754b484c', 'ethKeyName': '[SGX_KEY]', 'secretShare': '362c31d4a68dfcb02935ddf8d51ba442c7a3414dd2d91f524531afc0e01e8a1bfd5dae65a6aa055ba1597dfbe5f00ba113ee0706f3fa9efcc1617d2f6bc530ee419ce341b132d05c211e3c3bd3e684d0a51ce65652750751d6bb26019c094e10', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 17:20:08,543 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 17:20:08,543 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 11 was received and verified [2023-09-07 17:20:08,544 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 11 [2023-09-07 17:20:08,544 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 14 [2023-09-07 17:20:08,545 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:137 - sChain free-basic-rosy-SKALE: total received 8 broadcasts from nodes [9, 25, 61, 15, 35, 19, 66, 16] [2023-09-07 17:20:10,561 INFO][30673][cleaner][MainThread] - core.schains.cleaner:116 - sChains on contracts: ['portly-passionate-sirius', 'curly-red-alterf', 'howling-hoedus', 'green-giddy-denebola', 'wan-tasty-kitalpha', 'free-basic-rosy-SKALE', 'submissive-well-off-rotanev'] sChains on node: ['curly-red-alterf', 'free-basic-rosy-SKALE', 'green-giddy-denebola', 'howling-hoedus', 'portly-passionate-sirius', 'submissive-well-off-rotanev', 'wan-tasty-kitalpha'] [2023-09-07 17:20:10,563 INFO][30673][cleaner][MainThread] - core.schains.cleaner:132 - Cleanup procedure finished [2023-09-07 17:20:10,582 INFO][1][MainProcess][MainThread] - core.schains.cleaner:67 - Cleaner process is joined. [2023-09-07 17:20:10,583 INFO][1][MainProcess][MainThread] - core.schains.cleaner:68 - Terminating the process [2023-09-07 17:20:10,583 INFO][1][MainProcess][MainThread] - __main__:70 - Sleeping for 90s after run_cleaner [2023-09-07 17:20:15,488 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107200, schain_finish_ts: 1692637668 [2023-09-07 17:20:20,125 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107212, schain_finish_ts: 1693742304 [2023-09-07 17:20:22,184 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:20:22,185 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:20:22,185 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:20:22,185 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:20:22,185 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:20:22,188 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:20:22,206 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:20:22,379 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:20:22,631 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:20:22,656 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107212, schain_finish_ts: 1693742004 [2023-09-07 17:20:22,749 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:20:22,751 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2514 [2023-09-07 17:20:22,751 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:20:22,752 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:20:22,752 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:20:22,752 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:20:22,752 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:20:22,758 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:20:22,760 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:20:22.760765 [2023-09-07 17:20:22,893 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:20:22,932 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:20:23,017 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:20:23,020 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:20:23,030 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:20:23,031 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694107222.9265137, 'te': 1694107222.9388025, 'initial_status': True} ====================================================================================================  [2023-09-07 17:20:23,034 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:20:23.034039 [2023-09-07 17:20:23,039 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:20:23,040 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:20:24,471 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:20:24,472 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:20:24,472 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:20:24,472 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:20:24,473 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:20:24,473 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:20:24,477 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:20:24,479 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 17:20:24,506 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:20:24,681 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:20:24,815 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:20:24,932 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:20:24,934 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2508 [2023-09-07 17:20:24,934 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:20:24,935 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:20:24,935 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:20:24,935 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:20:24,935 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:20:24,942 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:20:24,944 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:20:24.944341 [2023-09-07 17:20:25,076 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:20:25,104 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:20:25,130 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107212, schain_finish_ts: 0 [2023-09-07 17:20:25,158 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:20:25,165 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:20:25,177 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:20:25,177 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694107225.102451, 'te': 1694107225.1091511, 'initial_status': True} ====================================================================================================  [2023-09-07 17:20:25,180 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:20:25.180118 [2023-09-07 17:20:25,187 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:20:25,188 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:20:27,817 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107224, schain_finish_ts: 1693412520 [2023-09-07 17:20:31,504 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 14 -> 11 [2023-09-07 17:20:40,439 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:104 - sChain free-basic-rosy-SKALE: trying to receive broadcasted data,1176 seconds left [2023-09-07 17:20:41,831 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:20:41,832 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:20:41,834 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 17:20:47,113 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:20:47,113 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:20:47,113 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:20:47,113 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:20:47,114 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:20:47,114 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:20:47,151 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:20:47,421 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:20:47,705 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:20:47,841 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:20:47,842 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2506 [2023-09-07 17:20:47,843 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:20:47,844 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:20:47,844 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:20:47,844 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:20:47,845 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:20:47,852 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:20:47,856 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:20:47.856254 [2023-09-07 17:20:48,049 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:20:48,092 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:20:48,137 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:20:48,139 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:20:48,152 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:20:48,152 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694107248.089772, 'te': 1694107248.0985885, 'initial_status': True} ====================================================================================================  [2023-09-07 17:20:48,155 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:20:48.155174 [2023-09-07 17:20:48,160 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:20:48,161 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:20:48,466 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107236, schain_finish_ts: 1692377364 [2023-09-07 17:20:48,501 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:20:48,502 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:20:48,502 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:20:48,502 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:20:48,502 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:20:48,502 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:20:48,505 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:20:48,509 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 17:20:48,542 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:20:48,741 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:20:48,741 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:20:48,743 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:20:48.743304 [2023-09-07 17:20:48,749 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:20:48,750 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:20:48,757 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:20:48,763 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694107248.7484126, 'te': 1694107248.7635133, 'initial_status': True} ====================================================================================================  [2023-09-07 17:20:48,766 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:20:48.766336 [2023-09-07 17:20:48,771 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:20:48,771 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:20:48,892 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:20:49,118 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:20:49,283 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:20:49,285 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2510 [2023-09-07 17:20:49,286 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:20:49,286 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:20:49,287 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:20:49,287 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:20:49,287 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:20:49,293 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:20:49,295 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:20:49.295313 [2023-09-07 17:20:49,425 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:20:49,486 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:20:49,518 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:20:49,520 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:20:49,527 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:20:49,528 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694107249.4849901, 'te': 1694107249.4932067, 'initial_status': True} ====================================================================================================  [2023-09-07 17:20:49,529 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:20:49.529717 [2023-09-07 17:20:49,535 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:20:49,536 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:20:51,278 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107236, schain_finish_ts: 1694108304 [2023-09-07 17:20:52,135 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.broadcast_filter:94 - sChain 0x000002bb9043962a56bacebbde03191f32a67a4b888426ac09f83548ae92e235: Parsing broadcast events from 9653867 block to 9653907 block [2023-09-07 17:20:57,965 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107248, schain_finish_ts: 1692637668 [2023-09-07 17:21:02,147 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:21:02,148 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:21:02,148 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 17:21:05,437 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:21:05,438 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:21:05,440 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:21:05.440650 [2023-09-07 17:21:05,449 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:21:05,449 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:21:05,450 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:21:05,450 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:21:05,451 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:21:05,451 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:21:05,452 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:21:05,452 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:21:05,455 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:21:05,455 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:21:05,457 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:21:05,463 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694107265.4476614, 'te': 1694107265.463198, 'initial_status': True} ====================================================================================================  [2023-09-07 17:21:05,465 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:21:05.465273 [2023-09-07 17:21:05,469 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:21:05,470 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:21:05,491 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:21:05,716 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:21:05,873 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:21:05,994 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:21:05,995 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2494 [2023-09-07 17:21:05,996 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:21:05,996 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:21:05,997 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:21:05,997 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:21:05,997 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:21:06,004 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:21:06,006 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:21:06.006762 [2023-09-07 17:21:06,183 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:21:06,212 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:21:06,250 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:21:06,255 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:21:06,268 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:21:06,272 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694107266.209851, 'te': 1694107266.217672, 'initial_status': True} ====================================================================================================  [2023-09-07 17:21:06,277 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:21:06.277512 [2023-09-07 17:21:06,284 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:21:06,288 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:21:09,472 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:21:09,473 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:21:09,473 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:21:09,473 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:21:09,473 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:21:09,474 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:21:09,476 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:21:09,479 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 17:21:09,515 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:21:09,728 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:21:10,078 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:21:10,183 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:21:10,184 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2509 [2023-09-07 17:21:10,184 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:21:10,185 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:21:10,185 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:21:10,185 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:21:10,186 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:21:10,191 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:21:10,193 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:21:10.193683 [2023-09-07 17:21:10,334 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:21:10,364 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:21:10,393 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:21:10,395 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:21:10,402 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:21:10,402 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694107270.3623488, 'te': 1694107270.3683534, 'initial_status': True} ====================================================================================================  [2023-09-07 17:21:10,404 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:21:10.404584 [2023-09-07 17:21:10,409 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:21:10,410 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:21:18,473 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107272, schain_finish_ts: 1693742304 [2023-09-07 17:21:34,128 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107272, schain_finish_ts: 0 [2023-09-07 17:21:35,578 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:21:35,578 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:21:35,581 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:21:35.581112 [2023-09-07 17:21:35,590 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:21:35,590 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:21:35,596 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:21:35,602 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694107295.5882742, 'te': 1694107295.602409, 'initial_status': True} ====================================================================================================  [2023-09-07 17:21:35,605 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:21:35.605715 [2023-09-07 17:21:35,612 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:21:35,612 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:21:38,551 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:21:38,551 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:21:38,551 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:21:38,552 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:21:38,552 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:21:38,552 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:21:38,573 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:21:38,762 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:21:38,904 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:21:39,027 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:21:39,028 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2515 [2023-09-07 17:21:39,028 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:21:39,029 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:21:39,029 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:21:39,029 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:21:39,029 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:21:39,035 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:21:39,037 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:21:39.037460 [2023-09-07 17:21:39,173 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:21:39,202 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:21:39,264 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:21:39,266 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:21:39,274 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:21:39,275 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694107299.200658, 'te': 1694107299.2086568, 'initial_status': True} ====================================================================================================  [2023-09-07 17:21:39,277 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:21:39.277177 [2023-09-07 17:21:39,282 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:21:39,283 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:21:40,618 INFO][1][MainProcess][MainThread] - core.schains.process_manager:57 - Process manager started [2023-09-07 17:21:40,907 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107296, schain_finish_ts: 1693412520 [2023-09-07 17:21:43,758 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:21:43,759 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:21:43,761 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:21:43.761128 [2023-09-07 17:21:43,767 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:21:43,768 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:21:43,773 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:21:43,778 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694107303.7664902, 'te': 1694107303.7778583, 'initial_status': True} ====================================================================================================  [2023-09-07 17:21:43,779 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:21:43.779736 [2023-09-07 17:21:43,784 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:21:43,784 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:21:44,294 INFO][1][MainProcess][MainThread] - core.schains.notifications:35 - Node account has 406909841189497969 WEI [2023-09-07 17:21:44,298 INFO][1][MainProcess][MainThread] - tools.notifications.messages:188 - Saving new balance state 3011 1 [2023-09-07 17:21:44,299 INFO][1][MainProcess][MainThread] - core.schains.process_manager:95 - Fetching schains to monitor... [2023-09-07 17:21:57,230 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107308, schain_finish_ts: 1693742004 [2023-09-07 17:22:04,221 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 56 -> 60 [2023-09-07 17:22:06,807 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:22:06,807 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:22:06,809 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:22:06,810 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:22:06,810 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:22:06,810 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:22:06,815 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:22:06,819 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:22:06,857 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:22:07,122 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:22:07,295 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:22:07,412 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:22:07,413 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2511 [2023-09-07 17:22:07,414 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:22:07,414 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:22:07,414 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:22:07,415 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:22:07,415 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:22:07,422 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:22:07,424 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:22:07.424727 [2023-09-07 17:22:07,605 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:22:07,656 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:22:07,699 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:22:07,701 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:22:07,710 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:22:07,710 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694107327.6531117, 'te': 1694107327.6621778, 'initial_status': True} ====================================================================================================  [2023-09-07 17:22:07,712 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:22:07.712683 [2023-09-07 17:22:07,718 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:22:07,718 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:22:09,697 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107320, schain_finish_ts: 1692377364 [2023-09-07 17:22:09,701 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:22:09,703 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:22:09,703 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 17:22:09,850 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107320, schain_finish_ts: 1694108304 [2023-09-07 17:22:18,539 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107332, schain_finish_ts: 1692637668 [2023-09-07 17:22:21,451 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:22:21,452 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:22:21,452 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:22:34,634 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:22:34,634 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:22:34,634 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:22:34,635 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:22:34,638 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:22:34,638 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:22:34,668 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:22:34,935 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:22:35,156 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:22:35,271 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:22:35,273 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2507 [2023-09-07 17:22:35,274 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:22:35,274 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:22:35,275 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:22:35,275 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:22:35,276 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:22:35,284 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:22:35,287 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:22:35.287656 [2023-09-07 17:22:35,480 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:22:35,504 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:22:35,531 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:22:35,533 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:22:35,540 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:22:35,540 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694107355.5023534, 'te': 1694107355.5089629, 'initial_status': True} ====================================================================================================  [2023-09-07 17:22:35,541 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:22:35.541929 [2023-09-07 17:22:35,545 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:22:35,546 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:22:43,994 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107356, schain_finish_ts: 1693742304 [2023-09-07 17:22:50,155 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:22:50,155 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:22:50,156 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:22:50,156 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:22:50,156 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:22:50,156 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:22:50,162 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:22:50,165 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 17:22:50,191 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:22:50,507 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:22:50,642 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:22:50,775 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:22:50,776 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2495 [2023-09-07 17:22:50,777 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:22:50,777 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:22:50,778 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:22:50,778 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:22:50,778 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:22:50,784 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:22:50,787 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:22:50.787072 [2023-09-07 17:22:50,961 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:22:50,989 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:22:51,022 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:22:51,025 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:22:51,032 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:22:51,033 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694107370.9876392, 'te': 1694107370.9942665, 'initial_status': True} ====================================================================================================  [2023-09-07 17:22:51,034 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:22:51.034926 [2023-09-07 17:22:51,039 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:22:51,040 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:23:03,140 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107380, schain_finish_ts: 1693412520 [2023-09-07 17:23:04,720 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:23:04,721 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:23:04,721 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:23:04,721 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:23:04,722 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:23:04,722 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:23:04,722 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:23:04,722 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 17:23:04,753 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:23:04,934 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:23:04,934 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:23:04,937 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:23:04.937194 [2023-09-07 17:23:04,949 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:23:04,949 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:23:04,956 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:23:04,962 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694107384.947153, 'te': 1694107384.9619439, 'initial_status': True} ====================================================================================================  [2023-09-07 17:23:04,964 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:23:04.964549 [2023-09-07 17:23:04,970 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:23:04,970 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:23:05,019 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:23:05,205 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:23:05,321 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:23:05,322 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2510 [2023-09-07 17:23:05,323 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:23:05,323 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:23:05,323 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:23:05,324 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:23:05,324 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:23:05,330 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:23:05,332 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:23:05.332837 [2023-09-07 17:23:05,477 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:23:05,503 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:23:05,531 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:23:05,533 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:23:05,540 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:23:05,540 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694107385.50175, 'te': 1694107385.5074897, 'initial_status': True} ====================================================================================================  [2023-09-07 17:23:05,542 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:23:05.542312 [2023-09-07 17:23:05,546 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:23:05,547 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:23:09,004 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107380, schain_finish_ts: 0 [2023-09-07 17:23:13,221 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:23:13,222 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:23:13,222 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:23:13,222 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:23:13,222 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:23:13,223 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:23:13,250 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:23:13,435 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:23:13,446 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107392, schain_finish_ts: 1694108304 [2023-09-07 17:23:13,685 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:23:13,795 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:23:13,796 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2516 [2023-09-07 17:23:13,797 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:23:13,798 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:23:13,798 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:23:13,798 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:23:13,799 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:23:13,806 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:23:13,808 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:23:13.808901 [2023-09-07 17:23:13,949 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:23:13,977 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:23:14,010 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:23:14,012 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:23:14,020 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:23:14,020 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694107393.97503, 'te': 1694107393.9823492, 'initial_status': True} ====================================================================================================  [2023-09-07 17:23:14,024 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:23:14.024667 [2023-09-07 17:23:14,029 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:23:14,029 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:23:17,302 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 17:23:17,302 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 17:23:22,909 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:23:22,909 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:23:22,909 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:23:22,909 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:23:22,910 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:23:22,910 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:23:22,912 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:23:22,914 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 17:23:22,937 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:23:23,226 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:23:23,360 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:23:23,480 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:23:23,481 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2512 [2023-09-07 17:23:23,482 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:23:23,482 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:23:23,482 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:23:23,483 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:23:23,483 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:23:23,489 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:23:23,492 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:23:23.492366 [2023-09-07 17:23:23,666 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:23:23,696 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:23:23,739 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:23:23,744 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:23:23,760 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:23:23,763 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694107403.6942108, 'te': 1694107403.7007394, 'initial_status': True} ====================================================================================================  [2023-09-07 17:23:23,773 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:23:23.772973 [2023-09-07 17:23:23,781 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:23:23,783 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:23:30,218 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107404, schain_finish_ts: 1693742004 [2023-09-07 17:23:30,276 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:23:30,276 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:23:30,277 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 17:23:42,626 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:23:42,627 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:23:42,632 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:23:44,948 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:23:44,949 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:23:44,952 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:23:44.952646 [2023-09-07 17:23:44,964 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:23:44,964 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:23:44,972 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:23:44,977 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694107424.9620652, 'te': 1694107424.9771247, 'initial_status': True} ====================================================================================================  [2023-09-07 17:23:44,979 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:23:44.979158 [2023-09-07 17:23:44,983 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:23:44,983 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:23:50,324 INFO][1][MainProcess][MainThread] - core.schains.process_manager:109 - Get leaving_history for node ... [2023-09-07 17:23:53,465 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:23:53,466 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:23:53,466 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:23:53,466 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:23:53,466 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:23:53,467 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:23:53,519 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:23:53,731 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:23:54,051 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:23:54,153 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:23:54,154 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2508 [2023-09-07 17:23:54,155 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:23:54,156 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:23:54,156 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:23:54,156 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:23:54,157 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:23:54,165 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:23:54,167 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:23:54.167597 [2023-09-07 17:23:54,323 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:23:54,349 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:23:54,379 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:23:54,381 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:23:54,389 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:23:54,389 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694107434.3479552, 'te': 1694107434.3545935, 'initial_status': True} ====================================================================================================  [2023-09-07 17:23:54,391 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:23:54.391188 [2023-09-07 17:23:54,395 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:23:54,396 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:24:06,462 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107440, schain_finish_ts: 1692377364 [2023-09-07 17:24:09,359 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:24:09,359 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:24:09,365 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:24:09.364989 [2023-09-07 17:24:09,373 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:24:09,373 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:24:09,379 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:24:09,384 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694107449.3718255, 'te': 1694107449.383912, 'initial_status': True} ====================================================================================================  [2023-09-07 17:24:09,387 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:24:09.387938 [2023-09-07 17:24:09,393 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:24:09,393 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:24:11,580 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107440, schain_finish_ts: 1693742304 [2023-09-07 17:24:11,581 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107440, schain_finish_ts: 1692637668 [2023-09-07 17:24:23,470 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107452, schain_finish_ts: 1693412520 [2023-09-07 17:24:24,792 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:24:24,792 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:24:24,794 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:24:24.794599 [2023-09-07 17:24:24,801 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:24:24,801 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:24:24,809 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:24:24,817 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694107464.799729, 'te': 1694107464.817599, 'initial_status': True} ====================================================================================================  [2023-09-07 17:24:24,819 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:24:24.819672 [2023-09-07 17:24:24,824 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:24:24,824 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:24:28,302 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:24:28,303 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:24:28,303 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:24:28,303 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:24:28,303 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:24:28,304 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 17:24:28,325 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:24:28,530 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:24:28,670 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:24:28,776 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:24:28,777 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2517 [2023-09-07 17:24:28,777 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:24:28,778 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:24:28,778 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:24:28,778 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:24:28,778 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:24:28,784 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:24:28,786 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:24:28.786820 [2023-09-07 17:24:28,943 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:24:28,967 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:24:28,993 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:24:28,995 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:24:29,002 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:24:29,003 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694107468.96569, 'te': 1694107468.9717703, 'initial_status': True} ====================================================================================================  [2023-09-07 17:24:29,004 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:24:29.004550 [2023-09-07 17:24:29,008 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:24:29,009 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:24:33,175 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107464, schain_finish_ts: 1694108304 [2023-09-07 17:24:33,404 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107464, schain_finish_ts: 1692377616 [2023-09-07 17:24:34,057 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:24:34,057 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:24:34,057 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:24:34,057 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:24:34,058 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:24:34,058 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:24:34,059 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:24:34,061 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 17:24:34,088 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:24:34,365 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:24:34,494 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:24:34,617 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:24:34,618 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2496 [2023-09-07 17:24:34,619 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:24:34,619 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:24:34,619 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:24:34,620 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:24:34,620 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:24:34,626 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:24:34,628 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:24:34.628511 [2023-09-07 17:24:34,766 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:24:34,795 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:24:34,824 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:24:34,827 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:24:34,837 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:24:34,837 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694107474.7930715, 'te': 1694107474.8005102, 'initial_status': True} ====================================================================================================  [2023-09-07 17:24:34,840 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:24:34.840031 [2023-09-07 17:24:34,845 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:24:34,846 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:24:36,764 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107464, schain_finish_ts: 1693742004 [2023-09-07 17:24:37,767 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107476, schain_finish_ts: 0 [2023-09-07 17:24:39,849 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:24:39,849 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:24:39,849 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:24:39,850 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:24:39,850 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:24:39,850 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:24:39,854 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:24:39,856 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 17:24:39,877 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:24:40,056 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:24:40,298 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:24:40,436 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:24:40,447 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2511 [2023-09-07 17:24:40,449 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:24:40,452 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:24:40,452 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:24:40,453 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:24:40,457 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:24:40,468 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:24:40,473 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:24:40.473908 [2023-09-07 17:24:40,657 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:24:40,683 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:24:40,714 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:24:40,716 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:24:40,724 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:24:40,724 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694107480.6817338, 'te': 1694107480.6893072, 'initial_status': True} ====================================================================================================  [2023-09-07 17:24:40,727 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:24:40.727074 [2023-09-07 17:24:40,733 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:24:40,734 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:24:56,461 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:24:56,461 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:24:56,462 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:24:56,462 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:24:56,462 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:24:56,463 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:24:56,468 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:24:56,475 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 17:24:56,512 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:24:56,926 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:24:57,093 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:24:57,191 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:24:57,192 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2513 [2023-09-07 17:24:57,192 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:24:57,193 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:24:57,193 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:24:57,193 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:24:57,194 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:24:57,202 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:24:57,204 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:24:57.204030 [2023-09-07 17:24:57,435 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:24:57,461 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:24:57,491 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:24:57,493 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:24:57,502 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:24:57,503 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694107497.459702, 'te': 1694107497.467037, 'initial_status': True} ====================================================================================================  [2023-09-07 17:24:57,504 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:24:57.504715 [2023-09-07 17:24:57,509 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:24:57,510 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:24:59,596 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107488, schain_finish_ts: 0 [2023-09-07 17:24:59,736 ERROR][24912][submissive-well-off-rotanev][T_0] - core.schains.task:26 - Task submissive-well-off-rotanev-config failed with max() arg is an empty sequence Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 63, in execute self.am.upstream_config() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 194, in upstream_config new_config = create_new_upstream_config( File "/usr/src/admin/core/schains/config/main.py", line 51, in create_new_upstream_config schain_config = generate_schain_config_with_skale( File "/usr/src/admin/core/schains/config/generator.py", line 213, in generate_schain_config_with_skale node_groups = get_previous_schain_groups(skale, schain_name) File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 56, in get_previous_schain_groups _add_previous_schain_rotations_state( File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 119, in _add_previous_schain_rotations_state new_node_id = max(previous_nodes.items(), key=lambda x: x[1]['finish_ts'])[0] ValueError: max() arg is an empty sequence [2023-09-07 17:24:59,736 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:25:02,203 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:25:02,203 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:25:02,204 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:25:02,205 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:25:02,205 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:25:02,205 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:25:02,206 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:25:02,207 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:25:02,207 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:25:02,241 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:25:02,429 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:25:02,559 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:25:02,679 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:25:02,681 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2509 [2023-09-07 17:25:02,682 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:25:02,682 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:25:02,683 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:25:02,683 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:25:02,683 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:25:02,691 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:25:02,693 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:25:02.693892 [2023-09-07 17:25:02,952 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:25:02,985 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:25:03,039 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:25:03,043 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:25:03,052 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:25:03,052 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694107502.9795787, 'te': 1694107502.9954119, 'initial_status': True} ====================================================================================================  [2023-09-07 17:25:03,053 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:25:03.053890 [2023-09-07 17:25:03,060 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:25:03,060 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:25:10,556 INFO][1][MainProcess][MainThread] - core.schains.process_manager:117 - Got leaving sChains for the node: [] [2023-09-07 17:25:10,557 INFO][1][MainProcess][MainThread] - core.schains.process_manager:101 -  ==================================================================================================== Monitoring sChains Node ID: 16 sChains on node: [{'name': 'portly-passionate-sirius', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666281384, 'startBlock': 7803380, 'deposit': 0, 'index': 1, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x0000017723ca2', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'curly-red-alterf', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1676058936, 'startBlock': 8469646, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000001fe6cdfa', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'howling-hoedus', 'mainnetOwner': '0x1026467C91fdA40aCc7e6935d1E52f67274FD669', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666181400, 'startBlock': 7796660, 'deposit': 0, 'index': 0, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0xe5a600f4c2608', 'multitransactionMode': False, 'thresholdEncryption': False, 'active': True}, {'name': 'green-giddy-denebola', 'mainnetOwner': '0x8e835d5d9E28ffD259E8Fa8988Cacfc333A582B4', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674565164, 'startBlock': 8368049, 'deposit': 0, 'index': 4, 'generation': 1, 'originator': '0xd2001DAb6898127Be2F167B548691C87251D13C3', 'chainId': '0x00000585eb4b1', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'wan-tasty-kitalpha', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674568392, 'startBlock': 8368271, 'deposit': 0, 'index': 5, 'generation': 1, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x00000152e7be9', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'free-basic-rosy-SKALE', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1693996920, 'startBlock': 9646443, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000002bb90439', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'submissive-well-off-rotanev', 'mainnetOwner': '0x487360568fE5Ee4AE5c75782D5F63e9935B0E512', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1692361104, 'startBlock': 9539053, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000003791fa10', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}] Number of sChains on node: 7 Empty sChain structs: 0 ====================================================================================================  [2023-09-07 17:25:14,069 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:25:14,070 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:25:14,070 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:25:16,199 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107500, schain_finish_ts: 1693742304 [2023-09-07 17:25:29,581 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 17:25:29,766 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:25:29,770 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:25:29,774 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:25:29,775 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:25:29,958 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:25:29,961 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:25:29,971 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 17:25:29,983 ERROR][11][wan-tasty-kitalpha][T_0] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:25:29,984 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:25:30,556 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:25:30,557 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain portly-passionate-sirius - Process is running: PID = 7 [2023-09-07 17:25:37,605 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107524, schain_finish_ts: 1692377364 [2023-09-07 17:25:38,422 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:25:38,424 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain curly-red-alterf - Process is running: PID = 8 [2023-09-07 17:25:39,752 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:25:39,752 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:25:39,754 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:25:39.754201 [2023-09-07 17:25:39,761 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:25:39,761 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:25:39,768 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:25:39,773 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694107539.760062, 'te': 1694107539.773031, 'initial_status': True} ====================================================================================================  [2023-09-07 17:25:39,774 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:25:39.774830 [2023-09-07 17:25:39,780 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:25:39,780 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:25:40,640 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:25:40,640 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:25:40,646 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:25:40.646530 [2023-09-07 17:25:40,659 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:25:40,660 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:25:40,670 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:25:40,676 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694107540.6572018, 'te': 1694107540.6756628, 'initial_status': True} ====================================================================================================  [2023-09-07 17:25:40,677 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:25:40.677632 [2023-09-07 17:25:40,682 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:25:40,682 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:25:40,779 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:25:40,780 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain howling-hoedus - Process is running: PID = 9 [2023-09-07 17:25:40,800 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:25:40,801 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:25:40,802 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:25:40.802903 [2023-09-07 17:25:40,811 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:25:40,812 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:25:40,818 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:25:40,824 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694107540.8100324, 'te': 1694107540.8241744, 'initial_status': True} ====================================================================================================  [2023-09-07 17:25:40,825 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:25:40.825936 [2023-09-07 17:25:40,831 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:25:40,832 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:25:40,841 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107536, schain_finish_ts: 1693412520 [2023-09-07 17:25:41,181 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:25:41,181 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain green-giddy-denebola - Process is running: PID = 10 [2023-09-07 17:25:41,568 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107536, schain_finish_ts: 1692637668 [2023-09-07 17:25:41,918 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:25:41,918 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain wan-tasty-kitalpha - Process is running: PID = 11 [2023-09-07 17:25:44,400 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107536, schain_finish_ts: 0 [2023-09-07 17:25:44,571 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:25:44,572 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:25:44,572 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:25:44,573 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:25:44,573 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:25:44,573 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:25:44,579 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:25:44,579 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:25:44,606 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:25:44,844 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:25:44,988 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:25:45,093 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:25:45,094 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2514 [2023-09-07 17:25:45,094 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:25:45,095 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:25:45,095 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:25:45,095 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:25:45,096 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:25:45,101 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:25:45,104 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:25:45.103979 [2023-09-07 17:25:45,248 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:25:45,278 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:25:45,307 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:25:45,308 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:25:45,316 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:25:45,316 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694107545.2768724, 'te': 1694107545.283003, 'initial_status': True} ====================================================================================================  [2023-09-07 17:25:45,318 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:25:45.318253 [2023-09-07 17:25:45,322 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:25:45,323 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:25:45,835 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:25:45,835 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:25:45,835 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:25:45,835 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:25:45,836 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:25:45,836 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:25:45,837 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:25:45,840 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:25:45,840 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain free-basic-rosy-SKALE - Process is running: PID = 30622 [2023-09-07 17:25:45,839 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:25:45,866 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:25:46,233 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:25:46,233 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:25:46,234 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:25:46,234 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 17:25:46,234 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 17:25:46,234 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:25:46,235 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:25:46,236 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:25:46,265 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:25:46,313 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:25:46,530 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:25:46,674 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:25:46,676 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2497 [2023-09-07 17:25:46,676 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:25:46,676 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:25:46,677 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:25:46,677 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:25:46,677 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:25:46,684 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:25:46,686 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:25:46.686048 [2023-09-07 17:25:46,771 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:25:46,912 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:25:46,947 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:25:46,996 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:25:46,998 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:25:47,009 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:25:47,010 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694107546.9454966, 'te': 1694107546.9528909, 'initial_status': True} ====================================================================================================  [2023-09-07 17:25:47,013 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:25:47.013453 [2023-09-07 17:25:47,021 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:25:47,022 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:25:47,340 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:25:47,492 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:25:47,494 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2518 [2023-09-07 17:25:47,495 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:25:47,495 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:25:47,495 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:25:47,496 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:25:47,496 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:25:47,504 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:25:47,507 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:25:47.507718 [2023-09-07 17:25:47,682 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:25:47,718 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:25:47,763 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:25:47,765 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:25:47,774 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:25:47,774 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694107547.7166975, 'te': 1694107547.7232785, 'initial_status': True} ====================================================================================================  [2023-09-07 17:25:47,776 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:25:47.776274 [2023-09-07 17:25:47,781 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:25:47,782 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:25:49,921 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107536, schain_finish_ts: 1693742004 [2023-09-07 17:25:50,183 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:25:50,183 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain submissive-well-off-rotanev - Process is running: PID = 24912 [2023-09-07 17:25:50,184 INFO][1][MainProcess][MainThread] - core.schains.process_manager:88 - Process manager procedure finished [2023-09-07 17:25:50,184 INFO][1][MainProcess][MainThread] - __main__:65 - Sleeping for 90s after run_process_manager [2023-09-07 17:25:55,690 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:25:55,690 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:25:55,690 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:25:55,691 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:25:55,691 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:25:55,691 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:25:55,694 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:25:55,697 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:25:55,725 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:25:55,977 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:25:55,977 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:25:55,981 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:25:55,981 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:25:55,981 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:25:55,982 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:25:56,045 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:25:56,049 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:25:56,290 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:25:56,529 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:25:56,531 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2512 [2023-09-07 17:25:56,531 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:25:56,532 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:25:56,532 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:25:56,532 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:25:56,533 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:25:56,540 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:25:56,542 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:25:56.542354 [2023-09-07 17:25:56,560 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:25:56,686 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:25:56,718 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:25:56,754 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:25:56,757 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:25:56,766 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:25:56,768 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694107556.7152607, 'te': 1694107556.723233, 'initial_status': True} ====================================================================================================  [2023-09-07 17:25:56,772 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:25:56.772879 [2023-09-07 17:25:56,778 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:25:56,779 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:25:56,828 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:25:56,948 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:25:56,949 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2510 [2023-09-07 17:25:56,950 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:25:56,950 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:25:56,950 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:25:56,951 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:25:56,951 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:25:56,957 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:25:56,959 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:25:56.959401 [2023-09-07 17:25:57,092 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:25:57,122 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:25:57,148 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:25:57,149 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:25:57,156 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:25:57,156 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694107557.1203086, 'te': 1694107557.126215, 'initial_status': True} ====================================================================================================  [2023-09-07 17:25:57,158 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:25:57.158106 [2023-09-07 17:25:57,162 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:25:57,162 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:25:59,505 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:25:59,508 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:25:59,509 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:40 - Task submissive-well-off-rotanev-config finished with None [2023-09-07 17:25:59,510 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:42 - Running task submissive-well-off-rotanev-config [2023-09-07 17:25:59,510 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 17:26:03,418 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:26:03,418 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:26:03,420 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:26:03.420461 [2023-09-07 17:26:03,427 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:26:03,428 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:26:03,433 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:26:03,437 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694107563.4265776, 'te': 1694107563.4372668, 'initial_status': True} ====================================================================================================  [2023-09-07 17:26:03,439 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:26:03.439729 [2023-09-07 17:26:03,444 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:26:03,444 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:26:03,636 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107560, schain_finish_ts: 1694108304 [2023-09-07 17:26:05,710 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:26:05,711 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:26:05,714 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:26:05.714628 [2023-09-07 17:26:05,722 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:26:05,723 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:26:05,730 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:26:05,735 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694107565.7206838, 'te': 1694107565.73491, 'initial_status': True} ====================================================================================================  [2023-09-07 17:26:05,737 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:26:05.737602 [2023-09-07 17:26:05,744 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:26:05,744 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:26:06,306 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:26:06,307 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:26:06,310 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:26:06.310931 [2023-09-07 17:26:06,320 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:26:18,893 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:26:18,894 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:26:18,894 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 17:26:21,019 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:26:21,020 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:26:21,022 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:26:21.022343 [2023-09-07 17:26:21,031 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:26:21,031 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:26:21,037 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:26:21,041 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694107581.0277503, 'te': 1694107581.0414262, 'initial_status': True} ====================================================================================================  [2023-09-07 17:26:21,043 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:26:21.043498 [2023-09-07 17:26:21,047 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:26:21,047 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:26:31,078 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107584, schain_finish_ts: 1693742304 [2023-09-07 17:26:33,851 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': False} [2023-09-07 17:26:33,851 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:26:33,853 INFO][24912][submissive-well-off-rotanev][T_0] - web.models.schain:152 - Changing monitor_last_seen for submissive-well-off-rotanev to 2023-09-07 17:26:33.853952 [2023-09-07 17:26:33,859 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.action:190 - Creating new upstream_config rotation_id: 8, stream: regression-2.2 [2023-09-07 17:26:33,861 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.config.main:49 - Generating sChain config for submissive-well-off-rotanev [2023-09-07 17:26:35,800 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107584, schain_finish_ts: 1692637668 [2023-09-07 17:26:39,419 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:26:39,420 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:26:39,420 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:26:39,421 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:26:39,422 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:26:39,422 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 17:26:39,523 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:26:39,689 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:26:39,825 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:26:39,920 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:26:39,921 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2519 [2023-09-07 17:26:39,922 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:26:39,922 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:26:39,922 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:26:39,923 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:26:39,923 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:26:39,929 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:26:39,931 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:26:39.931560 [2023-09-07 17:26:40,069 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:26:40,093 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:26:40,119 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:26:40,121 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:26:40,127 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:26:40,128 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694107600.0919719, 'te': 1694107600.0976212, 'initial_status': True} ====================================================================================================  [2023-09-07 17:26:40,129 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:26:40.129610 [2023-09-07 17:26:40,133 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:26:40,134 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:26:41,248 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107596, schain_finish_ts: 1692377364 [2023-09-07 17:26:41,255 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107596, schain_finish_ts: 0 [2023-09-07 17:26:41,339 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107596, schain_finish_ts: 1693742004 [2023-09-07 17:26:41,510 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107596, schain_finish_ts: 1693412520 [2023-09-07 17:26:48,742 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:26:48,742 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:26:48,743 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 17:26:48,918 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:26:48,918 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:26:48,918 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:26:48,918 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:26:48,919 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:26:48,919 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:26:48,921 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:26:48,923 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 17:26:48,953 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:26:49,145 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:26:49,276 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:26:49,375 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:26:49,376 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2498 [2023-09-07 17:26:49,377 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:26:49,377 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:26:49,377 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:26:49,378 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:26:49,378 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:26:49,384 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:26:49,386 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:26:49.386368 [2023-09-07 17:26:49,457 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:26:49,457 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:26:49,458 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:26:49,458 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:26:49,458 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:26:49,458 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:26:49,460 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:26:49,461 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:26:49,486 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:26:49,545 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:26:49,575 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:26:49,609 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:26:49,612 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:26:49,628 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:26:49,629 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694107609.5737965, 'te': 1694107609.5837185, 'initial_status': True} ====================================================================================================  [2023-09-07 17:26:49,631 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:26:49.631136 [2023-09-07 17:26:49,636 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:26:49,637 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:26:49,688 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:26:49,821 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:26:50,021 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:26:50,022 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2515 [2023-09-07 17:26:50,023 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:26:50,023 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:26:50,023 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:26:50,024 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:26:50,024 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:26:50,031 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:26:50,033 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:26:50.033571 [2023-09-07 17:26:50,169 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:26:50,202 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:26:50,233 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:26:50,235 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:26:50,244 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:26:50,244 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694107610.2010598, 'te': 1694107610.210329, 'initial_status': True} ====================================================================================================  [2023-09-07 17:26:50,246 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:26:50.246143 [2023-09-07 17:26:50,251 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:26:50,252 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:26:51,808 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:26:51,808 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:26:51,809 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:26:51,811 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:26:51,814 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:26:51,815 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:26:51,816 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:26:51,817 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:26:51,853 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:26:52,077 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:26:52,077 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:26:52,077 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:26:52,077 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:26:52,078 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:26:52,078 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:26:52,111 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:26:52,288 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:26:52,421 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:26:52,549 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:26:52,657 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:26:52,658 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2513 [2023-09-07 17:26:52,658 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:26:52,659 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:26:52,659 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:26:52,659 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:26:52,660 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:26:52,666 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:26:52,668 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:26:52.668578 [2023-09-07 17:26:52,674 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:26:52,804 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:26:52,805 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2511 [2023-09-07 17:26:52,805 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:26:52,805 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:26:52,806 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:26:52,806 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:26:52,806 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:26:52,807 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:26:52,812 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:26:52,814 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:26:52.814712 [2023-09-07 17:26:52,832 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:26:52,860 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:26:52,862 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:26:52,869 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:26:52,869 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694107612.8311024, 'te': 1694107612.8372285, 'initial_status': True} ====================================================================================================  [2023-09-07 17:26:52,871 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:26:52.871065 [2023-09-07 17:26:52,875 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:26:52,876 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:26:52,947 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:26:52,977 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:26:53,003 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:26:53,005 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:26:53,012 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:26:53,012 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694107612.9752927, 'te': 1694107612.9811914, 'initial_status': True} ====================================================================================================  [2023-09-07 17:26:53,014 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:26:53.014330 [2023-09-07 17:26:53,019 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:26:53,020 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:26:56,078 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107608, schain_finish_ts: 1694108304 [2023-09-07 17:27:00,717 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:27:00,717 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:27:00,721 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:27:00.721128 [2023-09-07 17:27:00,751 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:27:00,752 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:27:00,761 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:27:00,766 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694107620.742402, 'te': 1694107620.7658517, 'initial_status': True} ====================================================================================================  [2023-09-07 17:27:00,768 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:27:00.768873 [2023-09-07 17:27:00,773 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:27:00,774 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:27:06,126 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:27:06,126 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:27:06,128 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:27:06.128669 [2023-09-07 17:27:06,148 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:27:06,149 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:27:06,160 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:27:06,183 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694107626.1408615, 'te': 1694107626.183043, 'initial_status': True} ====================================================================================================  [2023-09-07 17:27:06,189 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:27:06.189433 [2023-09-07 17:27:06,196 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:27:06,198 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:27:08,694 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:27:08,695 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:27:08,697 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:27:08.697549 [2023-09-07 17:27:08,709 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:27:08,710 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:27:08,716 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:27:08,721 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694107628.7083325, 'te': 1694107628.721035, 'initial_status': True} ====================================================================================================  [2023-09-07 17:27:08,726 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:27:08.726648 [2023-09-07 17:27:08,732 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:27:08,732 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:27:08,875 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:27:08,878 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:27:08,880 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 17:27:14,411 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107608, schain_finish_ts: 1693412520 [2023-09-07 17:27:20,257 INFO][1][MainProcess][MainThread] - core.schains.cleaner:65 - Cleaner process started [2023-09-07 17:27:20,291 INFO][30722][cleaner][MainThread] - core.schains.cleaner:110 - Cleaner procedure started. [2023-09-07 17:27:25,024 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107632, schain_finish_ts: 1692377364 [2023-09-07 17:27:26,416 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:27:26,416 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:27:26,416 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:27:26,417 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:27:26,417 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:27:26,417 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:27:26,419 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:27:26,420 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 17:27:26,448 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:27:26,517 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107632, schain_finish_ts: 0 [2023-09-07 17:27:26,733 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:27:26,905 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:27:27,049 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:27:27,050 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2516 [2023-09-07 17:27:27,051 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:27:27,051 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:27:27,051 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:27:27,051 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:27:27,052 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:27:27,058 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:27:27,060 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:27:27.060238 [2023-09-07 17:27:27,204 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:27:27,230 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:27:27,268 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:27:27,270 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:27:27,287 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:27:27,288 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694107647.2279377, 'te': 1694107647.234354, 'initial_status': True} ====================================================================================================  [2023-09-07 17:27:27,290 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:27:27.290489 [2023-09-07 17:27:27,296 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:27:27,297 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:27:27,594 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:27:27,595 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:27:27,595 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:27:27,595 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:27:27,595 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:27:27,595 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:27:27,600 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:27:27,601 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:27:27,636 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:27:27,777 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107632, schain_finish_ts: 1693742004 [2023-09-07 17:27:27,948 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:27:28,085 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:27:28,188 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:27:28,189 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2499 [2023-09-07 17:27:28,190 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:27:28,190 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:27:28,191 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:27:28,192 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:27:28,192 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:27:28,199 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:27:28,201 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:27:28.201475 [2023-09-07 17:27:28,355 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:27:28,385 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:27:28,422 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:27:28,425 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:27:28,434 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:27:28,435 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694107648.3838627, 'te': 1694107648.390784, 'initial_status': True} ====================================================================================================  [2023-09-07 17:27:28,438 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:27:28.438379 [2023-09-07 17:27:28,443 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:27:28,444 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:27:29,287 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107632, schain_finish_ts: 1693742304 [2023-09-07 17:27:31,306 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 14 [2023-09-07 17:27:31,306 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 9 [2023-09-07 17:27:31,307 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '1f424cc357cd184791b648ce56290f2fec97be6e0e87e06939900a861000479d1b11dae749d05ae0e406525426c9505c1012b6a8ec281a7ff21ac7c5bd86393e0fa86513274d9ea9837ba5af9a7d4dba397f254f0f85e045dccc6130f919053b24f7e3979ba97e315ddc0525c2f632919c1ce7efc153eaa562e7a8fc20a0355f1ec4fb92ea51f35ffda2b850af79cbd792215bf3d12b756a09d866a40c8257a30bc91702f9327a545f43acaabc5a4a5aeb99afa941edd4d94da8a501b9060de32f69e7de0589143a7a4039c511bbf74f981d5d2ab8699b0c43051aadc0060eba1b929c38080933dd3e381eefef57849ef5f2f10f5f9580ab8717d7e0d61a70231cce84d27d6f0c97edbe3f3343849f409f5ac09bbf7ba2b07a33c74772f8385027988eb711bd3fc6e71fa8528f6eb62450b6ccc5f3b657eae454b3866dc9ac751ecf573831c2af7aea940c824a9e0478bd59672ae6c4c740adcdf914b0562c5311deb9169b068c75052e6b6bd0f1d96501ba61cfcaa1769d767ea5d490c5bdc317d08024083d36c1e3d170069fe6b2f9ab486e2aeff038a79085d9c8b7cf71950c447c6c1f9204f5957531c8228c56be18b53b53d22ac5e60e3368ef5c6f09b7295712f163144e0ca7fab8db6ef4dae2ccce7a24435dd9364802e134714505df02bd693ad2ce12d98a81969f47b787d75ad87a370a04423847d8b07390a9505a0ce63f288372b77f104ad93e96884fc2774b17c2ce03bc135fd48bd871902c1a1fb34954a204f76074870294d997344f1cd04141065c2337b3f476d44949fec42c9fbd268795030a36db3acc58a961e577f5a3ac6aea2d6dedddb5cba095a0ab1e98a203be9cfd123ca14aa0d2fdc6d4c1c7999e016760490caefdd6ae93ece626889ec2bc21b3664c5e43587ae7ea6c5bd5541afea7cf505e1932294a0ea66a1496441320d91451d2cc44787e7dc8ffa86e75e430acfccb4979c257d3d3fa1327e20c1892f68cd63a9b5ced3045e1c9e9517406f227fe3c606a62c2826ca6f42e2da972af12d250f134faa592f19ea6b7b53109117e8141ce88efd601d982302980ee344b52f487b2f07891a9545b2b55ba2d3e140acde2592ef7691873c13716850f90b23d84515b6ea6b16ee4bf26581537f78dc6417dde4608e86c5707e80650d862d4bc4a0b33ad29559a865db2cf4a7518e56792bd1244ce87ed95ca2e023e7cc9dea65a6557dc7c70b28d5956a3fa23d7c63dc2821ba292f72ba1c1402dfa2dc86fcc52f4f4d93bbd5f7503edad1457879644347b13296da12aa4646c2fe6a0892949944735b7f488a6c34078f3e9f3c4018022881d6b42d4afabeb0f1c6a74c730bbd720b0bb4f45a147251874b6ccdb35abfe85c1ca4c34a9e3946a082e95f64e19868bded66aa4b2140b8b2ea9da608c9dfcd5a440aff29acc24ab1ea6d7188bad31e2d4ae61021cc65d2f5b01d16644f534d1d527bb49d69e03bd22f44805ae6330f4dd61e5cd2ecee6b7b80fd5142a13800d563520fff53ed0e220c1868f03943f3f8f30dc3cf58bfb71a694a4ebefe173a58ebe990eef493acf2158650b8eb5198259af8938cc7bbd914947f228bc5c42679ce8493a51e330f323189e051d7237fc0b6c6d1accbf19aa82068334cf2369de969978a33823a006197a1215d59a8377925017ec8e33e43400858b2ca8f2a115fb8686e80d6c5354114b19b0d3c097f017442ef3047399da5467e9b928248e4c75730c9d12ba407b0a5380a5742adc306e06e48a76183ae7395b4d99486488da0fb78a8d1c29b43e01c946b31db90e396998aa75c1826906a03ac906eff11a74445bbd9600e730fc0e060fd5c529174bd2cc921b36f06ab2f467094033e9fedc7b66238f1b4ecb3f0000f9097f94ea1fbe53e706d49ca0946b3a70e9a5d99937aac571872bfbb6df22e0cd973465c05689c52ea2a4f4dc4268187bcb10e8ad6938ca99bb66def22e', 'ethKeyName': '[SGX_KEY]', 'secretShare': '7aa3274f03caa26dca6e50f39f4ae59d843b7bcf599ef89977c01e66b0a1e84466f1d583f496bc16d641910466fe56c5a57ec4566da10a81b7171665cc9b312e973942200f1610e1c797932336f10970107c8f9ddcb9fbd1108887da6e0ea5dc', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 17:27:31,524 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 17:27:31,524 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 9 was received and verified [2023-09-07 17:27:31,524 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 9 [2023-09-07 17:27:31,524 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 8 [2023-09-07 17:27:31,525 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '18036b23bff4c1e1c62715ff6d3a3b0d0371681ae650c91c4f1416d0b2120924016d5b6fabe06c8156a33ddf52c105530938c17422dc99d1007f0c8cbaa6bb38243e48f38c4ff74584ecad4ea8e66353dd432462518d2f369565c487b4531e810e67ac14fee9be3f5eddb38893b7c36cadde2163070827d4d8c879b4846e56bc01d3ea92933317160fc7354209b965bb41d3240e2c2e4b51fa627755fba1260a07c214ea666297727acc0ac351e7a85b4070ad364250d5bfc013cbbc295d9f632830769d305820fa6997fd81f145fd27063ab131d53bd901ee95efd4cfc922ab113a84cbb199159a8294fbf038cb3697019d74f38b942b3a853f3c4fa99d706a21e8072a7550d67de3947d547a9439af403e0a4da6c7748d9562ce9c2c32b9a319d2cccb63415309669cd9c9d0e59a15183e90fc8ebcde3e8f172126f63d1b9012c714cfbc9c64209676548b5e8e7e5128b4cf7bcbab4b9ee4220b91e9510dd624edce9267994c4b89b74a114b7c067c0111ba7255c384f058d65e83242e15062dea5146ad64fee45d15c8040a41f4f5cf5d8caed7080f797f0a9aa13af45b5b2521ec824551e6b82daddc8a632132a18340c65b2d548e0062b891ebc9aa8beb282af077d400b92a09687b59f3a88584a853916fbb995402a7f1ff2e86fec6bc138e0e62b010efc326a49399b89edd02d91df8e3f907252fa5648cb3e6c176692f4869a80e4a686c203a076260ca0ca341388b7f2efc2af774312bf7ae2d3e5c0fae384b12a06f8dd3254d41be9c72a84ddac0657d58d22fbc15b74f0b10660027b533c36f3a242950c8901442c4058cec532ce8025725de774b86d5225e394e0d0a78762f818113fe9cc7ea982ea056f65717a572a4717079dd17abc3ef56eb2092d3cefde960688abd268ef71dad0e8f867ab78328bb86c4a697fa0e21895d2d86193281ea8b8c1d71b7b9eb9be901d2b016af5ad6d0c511fb01c1d1c86cd225a301d348b3754c2afade0b1ff1ab01b32f6e4ae2db34d8cd602082a4c7245504accc0893adce850fe8ec2944ccaf64b4f4cc32b0d3844595ed2ba007a9b5750c9001c8337d7fa4972ed63f606a0925caeef822a3e2ba4ab910dcc42726f4ff2db2ebea7a1867bfe54751f51564f440ca7a8ad39ed3926ddf719ed9c1cac8f40a1372f6b74f19096ddd57bc5382f4a8b086bc671fe2e6ee76641e2860265009023237a6b8d69871a1fc23cb7bc9a2db4b1a62732d1b60eaaa0c1e8bbaaef61506e848b69f65740715cc0e653d0bbd8c93e77eb03a551bbab7e95a2e887949310056a3d16cea104c2c491bef0b113f0d901d2315f47c732693547af24a1944f315be290dd84c1278e27187dc1a2dae50263a4f5fa5bf1b982be936a3c7f874e40219e51b0cd88d4ec3363c04cbb5753774d36527b7c9e0ce94614803391000402b3bb4d756d1a610a6f170711dab056d3e61432137d8ed1626a114e5bfc076d309119d5e5c5f9c66850e0bff0614d2b02e608616d6f9f1644c5cc790fce354340d4503c8a96afd6651b66e2820eeb0a9228ee66c2a7ab1318641c922b2839df429940cef67e650d697ad75371b2b6ca3a727458c80ad1812456faa0306690b2c2675cdc700c61965df6f7dace845147c9f4561aa490147c226ef239123670e9215c75266cf577553158a01f42e86457910da61dd56ff8d918cce4fc3b20f1c1011ef8b67c923594e15fbe661a22e9ee76c7fe761f7c2edda4abd997c4e9cd5b52a8367e7ddce6b2071f257373ff632a48c12ef468ed97288167fd01c3f3d507b0d43a513a8732188ea880bf87e180317130b56fa3f6da6dbeb5041ec48d232862bd8d274707c29c257fc9f7b91fa6e18124692b6b900dd606267a2c143d5a207047cb3a37467e67f7a9b45ec465953794cdec9389f43c811c8e994509343ddf7095295e2cebe7a98b5ec88d650ea92617c1e580fb0ec25a82fb7692035d96dad', 'ethKeyName': '[SGX_KEY]', 'secretShare': 'e34c6daa2a23e90499fbbbc77a76c2693543b29eb5023a58e291cd34642a745544c423a5e7bcdc3c679d759678eecb621dfb7f8a63b5d5e9e71a828abad1c71525fa27fbe4ad5063e91ae1b284b770aacb53bc31ff5f03bea8e78eada0d8c0da', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 17:27:31,737 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 17:27:31,737 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 8 was received and verified [2023-09-07 17:27:31,738 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 8 [2023-09-07 17:27:31,738 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 10 [2023-09-07 17:27:31,738 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '2146e09b877649d3882c013c323937a5fdbe050f91bb14cb56b0b0867465bd131817635e0680e282c0a115a4869fd7bca3ad99d07516710eaa2f29aa8d7db5b70555483ad522dae8c6cb712e85f70f431c370c6026405b6994318f4e69d52b202836ae1fe0a94e9d1cc0028904a1868eeed3937d760c10b515255f03c337e2a01d58aa303eaab5ebcbbe95b50adf0feca885c85afa870649dfd1c63a015624032af9f3f865d3fce4493e180d17b9263f9809897cd3ddbef78fcd43021b03f5f52b837bcd87b6bbf142428b408e42e4e74e16f61e8852155ea312e54c6847d2f010a07e4ef165d29e508922bc87cba7174f5c912ffedb8f2e1b6bc6a368dd252922b96b008a7c174adc5b58a21f130b40e6504662adbdf4896d8116b8495c831f2d5b67012f278f6b5bb829fdbe217342f23b623090e187f7aaceb2d779cfe8b5168f3c9c1f36eb90aa51b1cbc0d668be5b5dfd418479f4d8e1b283443a1ddb690e130277a8c00dc3bf941b5704bcbf6aee94ac909f56eb55f0eb9b51c069369a199821629d79473231d45a59a3ea8fab45015f219e23e6da4c3e74fbaa69b67224df816fc44b0f7815b6df3dbe05e2c1bafa77daa321b5ff4a1192f09bec047b202f026f402a1057519a06db02c0745ae925d82f169c6f07cf9e50bcab60b8da13e44968e8a57f89ba1f9f7ec141874497282dcb466d7c0a4af63210bf64ce2315d357d9c026433d5fadc620336091de2a85be870d7315e71e2e765ddc0a77f20788f16f1f3c0b6f12ee45ba64aff2d14f9c96643c0dec760019010aa0dce83419643f8edd2259c9219e40f0846980808c0b29f322d1ba415d161efe80469a761e3b091010afe49498de6f00a42081b55ffa2ff1617481545f84ce077e4722332ab48dc1a231b846733839255d48c2afde0972ded0c6266e3c046f201cbc58f7290cdb25e33c4e86553aac99dd689b7b7fb82ee25220a3edfc4c0180b328bb4807a164a2e969137f08539b0ed6f27e91f15c28e9dc239f95362895ff0e1959c72b76318106daf18517de969136cb71d9c1b65d557122ce30dc2c2ce12279f68211c7001b5962166f814b0ecf91c26b74b8ee28727a2f3c8bfb6925706c1ba1af2e15f5f41664f77cfdc8c5064ca068f082a564e044e21f91e72fb6b5207a30d808f9c5e590016d0a0df1a36dc4baba1c7f3f73e944f4db52ba87d6daf33d796b10682ce2e4c56a35414fdd2e392bb46ab8c5f8101ae4cc68389414de1c426c5f2062585c777d063f539ba035ed1d6469d172b7ee40fd54608a2b547ddebd6c3b23cd338d56e2de83d306a8701658f479f9c2e6180030a60ca40a139b725085d32fba110c18d5bdab356ccda077cbab5b9974cbe44bf3a286573a6436561056671a74d0d21663f9594391046985b84d20fe448875c6b436ac36e5697fc30aa44521241a5322f4ac01491e6bad4fed2b73c074847a76b982b00c43c5dfd732171d05b590dfea7eaa7389af58802b06b503e5da0909a9d023ec84b527efbd8b535f07a9747479d908f39b2b667edaed6ae1642b1620a0a86ff4ddbe2bb40f42e0a20744e015fd6b20b064e515cd84b615db5f03265a0a3c193e09f85d0bc9736c022903b957e6a7a99c10f94407976df9a5061d2e56e6ad71c1e02f1bda857121b61d1325ee6962f2bffac8240382040211eadac16f7e19bee5d919eae4fc33f347250e68585c808f94a51f8ca53297d8f48bcc6e1c5877adb5b97d30674312d62524994a31d6b7f9ba32a6200fe650d71d5d5ada63b7cef55e9144d720498fb4ff06f48a5d5415ae9a557ad99990386495c0c6476f975117c20310d4eac44637cd1c313fb19537408161c2ceb12eb64b55c1977eb92add1b6b24105a0998f40443245ca74fb21fe6da13ab0783270b7c0bec22dd4018278fab8404edc8d4f442fd2d172d25c8ddd324e31c02fd85253949d690064ab86abc5b9893d0b39d214ce7', 'ethKeyName': '[SGX_KEY]', 'secretShare': '3e43e6ad8e821f46f82551aa031f8ec94e2f5136352f6e2b89fd83457ec0fd5435701bef715bbdb26bad52a1dfdce9798496af47df507e3b93365f27e3a2ce630c362524d225a71d67a928e12805f500d2536c74ad99621311b4006efeb50667', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 17:27:31,928 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 17:27:31,928 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 10 was received and verified [2023-09-07 17:27:31,928 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 10 [2023-09-07 17:27:31,929 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 15 [2023-09-07 17:27:31,929 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '1cdde40ee8aa26fb2832125b01f46de3b56bf7a2526a02a0e980ede57ed01b321bec5efeccc6c10fe674e3e87e2751e9a04f9ec3c8e9239588a293d67b0734852e353869fcbd8feb9d2b39d3eff9fdd9a65a506fb033c0f99b6a557aea69dad81f9d2f4b097d2aaa40b861a67ef1baba69d4ab8b839d667f079f88a477a3abb51d839f9de71d5b125942ddd7808b52a556918b659bc925d6b218e3490de72f121015942e0662254691c63e7ad66e986b9bf02a830f02d93468cfb2836482d58c2580e5e4e2f2e7a292ef2c7fa094eed5975e7af03c67c004e5700256482aef731dc74feff6928b1a74372f9fd21a10b351f8af3d2736f42156b4bd55f250de391217bd48cf6f4cbd7282b05347df97eccaee228711bca41d8b6aef03c1e793062722ebafbdeddf760395f841aecdd42cbd57d5def921ef66f3b903e6eba163290a243611070ce217e44db977759055dd163c2cd635da526ed61bee0159f65ac41ba0f7d6c2ba38feaf307a7b6dfb42c5cac1734c9a19490d4f4c87c51adeb37d015f9fc0463474076fd1f1e09e447980163072029daf2fe2f240be208af817751273558022940e88f67ca11307e68138ffac2d00c096ad24aa69d3a3a415461e0ab052c9faa86db130ff91d8208bd9fbb75001dceecb5e93077ea2f84b45b81f1545f69ba06ff7479b03f1ca5eda3a5db4abaa735484cb2314be36933110f4860dce9eb289b652680330752c067ebe75ebdfa037e347de663cfed6da9b9f47bb16bdf3775424e9f90d7f9436480a320067fdfe18e74f935f9a171f515f22e92b0b5c6844f614c46d3cc610648768ba71abfee266480c88c22721e7b4642ec2ca2be11597273d81b66fe06c2d552b644ac26a1f0cf29c075558cff3b7487c73e80b63ed661da81e234a7c308891c9874727351bd0d06268362c7cc73b27d7b5e30b6726b9eb4271e822865e908dafa053caa2eaebd9b0b89dcf835e88929de9df006ababd8b93b1e3de209919c45a2e0cd1b04978382d274ac27cbd7ad4c31e250defac22c7c8bfa7405e8b7600c853697d919a7d3e9126067efe136de41ac0f20738c4107859786367709b2a7ae8bb767e495f5e3e5ea2660589a6c3b5906c6d13380a9f23f3b487b83177cec965df303c72baba47bf58777cddeae36faba64812ea479b82f82e88b8c354a3c70740ed30c1ba35b8d1df146ea9f14041ab3e500b75f7c94210fb238ce275431ded93fbf31b22084b8a73ed93185180560c411f1d5821d3b85fb7c4a6c5c681231ecf5cbc13aec6370d29a51334bc5c71b89f6f1609bf538afa96246cb9d1d7a090e6260f8b2208eebb62d656e3c4545cd9c6882ea54e8fb848487f9d884dc471accb73357b620c4246ae5d47002c3c599ccffc060b0a6fd977d0906947630935ab8154124c3f0a30c83959f7e5d87abdfbaeac01b93057aab2b1f3c786adaf8ab61676b78be90e72846ccf873658f382827a280aae3e21e985ee4580f50e8e7979cd78fdeef30ec10eb9c319719d49d80ca981265b37ecafb536067b7e24603bddeeadad82fecb094493bc98021bb10ef01519209af18247af968bfe05088ffd5ca161b662c8a03ae311f3064539c951f759cc1efb946197111cf638e6d2696d00d8da3633b0247fbb7e19e42c95944a12c05f1560dd9333277c2ac6bee27a824c9bcdb71e0afbe87e0621ddc4477bd277e65a08aacc2fb934f981e7436b5fe8221d8df80022d444757fa4cf4b53f8a688c2192fa558abadeb511b187df7148d91f61b677da20bc3d4314eea874da14adc0a632bfcfe914dcfa7db031ba30e316872a052ae81310be29262e3e634cf88ee82530fa9e766012913f7bf85c8e1c1fd97e445cfed852dc79b7bf9266c43843fdb9b1b0f20999790fecf50f493b1974a09f0077909aacc276e50319ce701b79d1ce2087bcf675eccbaeb11000f24175901db58f6f7297191c769e669e102713d030b', 'ethKeyName': '[SGX_KEY]', 'secretShare': 'fc7939312f3db774ffbf98606c21174e6c07cfd122bc2bcc9a21cce900120b7bee5b2b11289ba6a4ef23f84eb629de6f4df5fd9a150f968e14412798b220538d755eaaca07e22232de6922e846c49b76c3dd8f0087197b322f011e703e1245c9', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 17:27:32,135 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 17:27:32,135 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 15 was received and verified [2023-09-07 17:27:32,136 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 15 [2023-09-07 17:27:32,136 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:137 - sChain free-basic-rosy-SKALE: total received 13 broadcasts from nodes [9, 25, 61, 15, 35, 19, 66, 16, 16, 24, 29, 23, 20] [2023-09-07 17:27:32,806 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:27:32,807 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:27:32,810 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:27:32.810760 [2023-09-07 17:27:32,818 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:27:32,819 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:27:32,828 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:27:32,833 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694107652.8166928, 'te': 1694107652.8327851, 'initial_status': True} ====================================================================================================  [2023-09-07 17:27:32,834 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:27:32.834650 [2023-09-07 17:27:32,839 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:27:32,839 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:27:34,593 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:27:34,593 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:27:34,594 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:27:34,650 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107632, schain_finish_ts: 1692637668 [2023-09-07 17:27:34,814 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:27:34,814 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:27:34,815 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:27:34,815 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:27:34,815 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:27:34,815 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 17:27:34,841 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:27:35,146 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:27:35,274 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:27:35,376 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:27:35,378 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2520 [2023-09-07 17:27:35,378 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:27:35,378 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:27:35,379 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:27:35,379 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:27:35,379 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:27:35,388 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:27:35,390 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:27:35.390745 [2023-09-07 17:27:35,536 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:27:35,568 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:27:35,608 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:27:35,611 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:27:35,620 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:27:35,621 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694107655.5664709, 'te': 1694107655.5745115, 'initial_status': True} ====================================================================================================  [2023-09-07 17:27:35,623 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:27:35.623634 [2023-09-07 17:27:35,628 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:27:35,629 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:27:36,149 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:27:36,151 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:27:36,153 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:27:36,155 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:27:36,157 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:27:36,157 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:27:36,180 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:27:36,580 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:27:36,712 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:27:36,833 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:27:36,835 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2512 [2023-09-07 17:27:36,835 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:27:36,836 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:27:36,836 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:27:36,836 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:27:36,837 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:27:36,845 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:27:36,848 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:27:36.848077 [2023-09-07 17:27:36,999 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:27:37,026 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:27:37,056 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:27:37,058 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:27:37,065 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:27:37,065 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694107657.0246663, 'te': 1694107657.0315921, 'initial_status': True} ====================================================================================================  [2023-09-07 17:27:37,067 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:27:37.067048 [2023-09-07 17:27:37,071 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:27:37,072 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:27:40,561 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:27:40,562 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:27:40,562 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:27:40,562 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:27:40,562 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:27:40,563 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:27:40,564 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:27:40,564 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:27:40,590 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:27:40,793 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:27:40,936 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:27:41,041 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:27:41,042 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2514 [2023-09-07 17:27:41,043 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:27:41,043 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:27:41,044 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:27:41,044 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:27:41,044 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:27:41,050 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:27:41,053 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:27:41.053049 [2023-09-07 17:27:41,211 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:27:41,241 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:27:41,246 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:27:41,247 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:27:41,249 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:27:41.249324 [2023-09-07 17:27:41,256 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:27:41,257 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:27:41,264 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:27:41,268 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694107661.2544842, 'te': 1694107661.2681549, 'initial_status': True} ====================================================================================================  [2023-09-07 17:27:41,270 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:27:41.270326 [2023-09-07 17:27:41,274 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:27:41,274 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:27:41,276 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:27:41,278 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:27:41,286 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:27:41,286 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694107661.2398314, 'te': 1694107661.2480502, 'initial_status': True} ====================================================================================================  [2023-09-07 17:27:41,289 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:27:41.289370 [2023-09-07 17:27:41,293 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:27:41,294 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:27:48,287 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:27:48,287 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:27:48,290 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:27:48.289976 [2023-09-07 17:27:48,298 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:27:48,298 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:27:48,304 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:27:48,309 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694107668.2961571, 'te': 1694107668.3090026, 'initial_status': True} ====================================================================================================  [2023-09-07 17:27:48,312 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:27:48.311996 [2023-09-07 17:27:48,317 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:27:48,317 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:27:50,108 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 17:27:50,282 INFO][30722][cleaner][MainThread] - core.schains.cleaner:116 - sChains on contracts: ['portly-passionate-sirius', 'curly-red-alterf', 'howling-hoedus', 'green-giddy-denebola', 'wan-tasty-kitalpha', 'free-basic-rosy-SKALE', 'submissive-well-off-rotanev'] sChains on node: ['curly-red-alterf', 'free-basic-rosy-SKALE', 'green-giddy-denebola', 'howling-hoedus', 'portly-passionate-sirius', 'submissive-well-off-rotanev', 'wan-tasty-kitalpha'] [2023-09-07 17:27:50,283 INFO][30722][cleaner][MainThread] - core.schains.cleaner:132 - Cleanup procedure finished [2023-09-07 17:27:50,287 INFO][1][MainProcess][MainThread] - core.schains.cleaner:67 - Cleaner process is joined. [2023-09-07 17:27:50,288 INFO][1][MainProcess][MainThread] - core.schains.cleaner:68 - Terminating the process [2023-09-07 17:27:50,288 INFO][1][MainProcess][MainThread] - __main__:70 - Sleeping for 90s after run_cleaner [2023-09-07 17:27:50,297 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:27:50,300 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:27:50,301 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:27:50,302 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:27:50,492 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:27:50,492 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:27:50,495 INFO][8][curly-red-alterf][T_1] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 17:27:50,502 ERROR][8][curly-red-alterf][T_1] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:27:50,502 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:27:58,617 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107668, schain_finish_ts: 1694108304 [2023-09-07 17:28:04,246 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107668, schain_finish_ts: 1693412520 [2023-09-07 17:28:10,446 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:326 - sChain: free-basic-rosy-SKALE. 14 node is trying to sent a complaint_bad_data on 3 node [2023-09-07 17:28:10,461 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:28:10,461 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:28:10,462 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:28:18,972 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107692, schain_finish_ts: 1693742304 [2023-09-07 17:28:19,233 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:28:19,234 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:28:19,234 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:28:19,234 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:28:19,234 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:28:19,235 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:28:19,237 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:28:19,237 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 17:28:19,266 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:28:19,343 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107692, schain_finish_ts: 1692377364 [2023-09-07 17:28:19,457 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:28:19,522 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107692, schain_finish_ts: 1692637668 [2023-09-07 17:28:19,738 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:28:19,835 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:28:19,836 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2517 [2023-09-07 17:28:19,837 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:28:19,837 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:28:19,837 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:28:19,837 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:28:19,838 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:28:19,846 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:28:19,848 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:28:19.848750 [2023-09-07 17:28:20,006 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:28:20,030 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:28:20,064 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:28:20,066 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:28:20,073 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:28:20,074 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694107700.029171, 'te': 1694107700.035958, 'initial_status': True} ====================================================================================================  [2023-09-07 17:28:20,075 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:28:20.075786 [2023-09-07 17:28:20,080 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:28:20,081 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:28:25,149 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107704, schain_finish_ts: 1693742004 [2023-09-07 17:28:28,162 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:28:28,162 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:28:28,162 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:28:28,162 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:28:28,163 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:28:28,163 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:28:28,166 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:28:28,167 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:28:28,196 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:28:28,211 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107704, schain_finish_ts: 0 [2023-09-07 17:28:28,484 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.transactions.tools:46 - Dry run tx: complaintBadData, sender: 0x523E13D962A7614752eFa3dB62e0466dF0d9E89D, wallet: RedisWalletAdapter, value: 0, [2023-09-07 17:28:28,500 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:28:28,501 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:28:28,501 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:28:28,501 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:28:28,501 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:28:28,502 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:28:28,503 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:28:28,506 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 17:28:28,513 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:28:28,535 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:28:28,637 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:28:28,640 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:28:28,645 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:28:28,646 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:28:28,651 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:28:28,652 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 17:28:28,674 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:28:28,712 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:28:28,829 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:28:28,830 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2515 [2023-09-07 17:28:28,831 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:28:28,831 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:28:28,832 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:28:28,832 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:28:28,832 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:28:28,840 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:28:28,842 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:28:28.842671 [2023-09-07 17:28:28,879 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:28:29,041 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:28:29,190 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:28:29,218 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:28:29,248 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:28:29,250 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:28:29,257 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:28:29,257 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694107709.2169843, 'te': 1694107709.2243285, 'initial_status': True} ====================================================================================================  [2023-09-07 17:28:29,259 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:28:29.259343 [2023-09-07 17:28:29,263 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:28:29,264 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:28:29,333 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:28:29,434 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:28:29,435 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2500 [2023-09-07 17:28:29,436 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:28:29,436 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:28:29,436 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:28:29,437 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:28:29,437 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:28:29,444 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:28:29,446 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:28:29.446743 [2023-09-07 17:28:29,476 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:28:29,597 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:28:29,598 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2521 [2023-09-07 17:28:29,599 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:28:29,599 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:28:29,599 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:28:29,600 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:28:29,600 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:28:29,609 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:28:29,611 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:28:29.611948 [2023-09-07 17:28:29,723 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:28:29,756 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:28:29,790 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:28:29,792 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:28:29,800 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:28:29,801 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694107709.7547426, 'te': 1694107709.7625864, 'initial_status': True} ====================================================================================================  [2023-09-07 17:28:29,802 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:28:29.802875 [2023-09-07 17:28:29,807 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:28:29,808 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:28:29,874 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:28:29,916 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:28:29,949 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:28:29,951 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:28:29,960 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:28:29,961 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694107709.913988, 'te': 1694107709.9227283, 'initial_status': True} ====================================================================================================  [2023-09-07 17:28:29,962 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:28:29.962974 [2023-09-07 17:28:29,974 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:28:29,975 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:28:30,999 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:28:31,000 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:28:31,000 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 17:28:32,260 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 17:28:32,260 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 17:28:32,321 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:28:32,322 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:28:32,322 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:28:32,322 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 17:28:32,322 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 17:28:32,323 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:28:32,323 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:28:32,324 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:28:32,348 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:28:32,525 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:28:32,664 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:28:32,760 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:28:32,761 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2513 [2023-09-07 17:28:32,762 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:28:32,762 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:28:32,762 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:28:32,762 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:28:32,763 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:28:32,769 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:28:32,771 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:28:32.771110 [2023-09-07 17:28:32,915 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:28:32,939 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:28:32,974 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:28:32,976 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:28:32,984 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:28:32,984 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694107712.937822, 'te': 1694107712.9442356, 'initial_status': True} ====================================================================================================  [2023-09-07 17:28:32,986 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:28:32.986514 [2023-09-07 17:28:32,991 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:28:32,992 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:28:38,620 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:28:38,621 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:28:38,625 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:28:38.625389 [2023-09-07 17:28:38,641 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:28:38,641 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:28:38,653 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:28:38,664 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694107718.635354, 'te': 1694107718.6620684, 'initial_status': True} ====================================================================================================  [2023-09-07 17:28:38,669 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:28:38.669106 [2023-09-07 17:28:38,675 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:28:38,675 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:28:39,280 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:28:39,280 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:28:39,284 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:28:39.284183 [2023-09-07 17:28:39,293 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:28:39,293 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:28:39,300 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:28:39,311 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694107719.291143, 'te': 1694107719.3111165, 'initial_status': True} ====================================================================================================  [2023-09-07 17:28:39,314 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:28:39.314189 [2023-09-07 17:28:39,320 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:28:39,320 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:28:40,143 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.transactions.tools:60 - Estimated gas for complaintBadData: 333375 [2023-09-07 17:28:40,209 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:28:40,209 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:28:40,211 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:28:40.211839 [2023-09-07 17:28:40,223 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:28:40,223 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:28:40,231 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:28:40,237 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694107720.220307, 'te': 1694107720.2374368, 'initial_status': True} ====================================================================================================  [2023-09-07 17:28:40,240 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:28:40.240346 [2023-09-07 17:28:40,245 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:28:40,246 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:28:40,271 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107716, schain_finish_ts: 1694108304 [2023-09-07 17:28:40,338 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.transactions.tools:126 - Tx: complaintBadData, Fields: {'chainId': 5, 'gas': 333375, 'nonce': 313, 'value': 0, 'gasPrice': 10000000000, 'type': 1, 'to': '0x69E2C9576B36c8d7938Ba45a1C909B67EC53048E', 'data': '0x2fce1ddf000002bb9043962a56bacebbde03191f32a67a4b888426ac09f83548ae92e2350000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000001f'}, [2023-09-07 17:28:40,339 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.wallets.redis_wallet:135 - Sending {'chainId': 5, 'gas': 333375, 'nonce': 313, 'value': 0, 'gasPrice': 10000000000, 'type': 1, 'to': '0x69E2C9576B36c8d7938Ba45a1C909B67EC53048E', 'data': '0x2fce1ddf000002bb9043962a56bacebbde03191f32a67a4b888426ac09f83548ae92e2350000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000001f'} to redis pool, method: dkg.complaintBadData [2023-09-07 17:28:40,339 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.wallets.redis_wallet:145 - Adding tx b'tx-2821ad628b96e840' to the pool [2023-09-07 17:28:40,340 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.wallets.redis_wallet:147 - Saving tx b'tx-2821ad628b96e840' record: b'{"status": "PROPOSED", "score": 51694107720, "multiplier": null, "tx_hash": null, "method": "dkg.complaintBadData", "meta": null, "chainId": 5, "gas": null, "nonce": 313, "value": 0, "gasPrice": 10000000000, "type": 1, "to": "0x69E2C9576B36c8d7938Ba45a1C909B67EC53048E", "data": "0x2fce1ddf000002bb9043962a56bacebbde03191f32a67a4b888426ac09f83548ae92e2350000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000001f"}' [2023-09-07 17:28:47,430 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:28:47,430 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:28:47,435 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:28:47.435662 [2023-09-07 17:28:47,442 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:28:56,168 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:28:56,169 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:28:56,170 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:29:05,941 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107716, schain_finish_ts: 1693412520 [2023-09-07 17:29:12,564 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107716, schain_finish_ts: 0 [2023-09-07 17:29:12,656 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107716, schain_finish_ts: 1692377364 [2023-09-07 17:29:12,767 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107716, schain_finish_ts: 1692637668 [2023-09-07 17:29:14,938 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107752, schain_finish_ts: 1693742304 [2023-09-07 17:29:19,574 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:29:19,575 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:29:19,575 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:29:19,575 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:29:19,576 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:29:19,576 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:29:19,577 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:29:19,581 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 17:29:19,611 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:29:19,813 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:29:19,958 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:29:20,115 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:29:20,117 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2516 [2023-09-07 17:29:20,117 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:29:20,117 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:29:20,118 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:29:20,118 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:29:20,118 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:29:20,124 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:29:20,126 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:29:20.126823 [2023-09-07 17:29:20,316 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:29:20,369 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:29:20,378 INFO][1][MainProcess][MainThread] - core.schains.process_manager:57 - Process manager started [2023-09-07 17:29:20,411 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:29:20,413 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:29:20,421 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:29:20,421 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694107760.3671575, 'te': 1694107760.375303, 'initial_status': True} ====================================================================================================  [2023-09-07 17:29:20,423 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:29:20.423745 [2023-09-07 17:29:20,429 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:29:20,431 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:29:21,549 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:29:21,550 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:29:21,550 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:29:21,550 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:29:21,551 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:29:21,551 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:29:21,553 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:29:21,558 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 17:29:21,558 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107752, schain_finish_ts: 1693742004 [2023-09-07 17:29:21,600 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:29:21,817 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:29:21,961 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:29:22,060 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:29:22,061 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2518 [2023-09-07 17:29:22,062 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:29:22,062 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:29:22,063 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:29:22,063 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:29:22,063 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:29:22,073 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:29:22,076 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:29:22.076000 [2023-09-07 17:29:22,211 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:29:22,241 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:29:22,271 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:29:22,273 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:29:22,282 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:29:22,282 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694107762.238416, 'te': 1694107762.247123, 'initial_status': True} ====================================================================================================  [2023-09-07 17:29:22,284 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:29:22.284278 [2023-09-07 17:29:22,290 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:29:22,291 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:29:22,450 INFO][1][MainProcess][MainThread] - core.schains.notifications:35 - Node account has 406903463189434189 WEI [2023-09-07 17:29:22,451 INFO][1][MainProcess][MainThread] - tools.notifications.messages:188 - Saving new balance state 3012 1 [2023-09-07 17:29:22,452 INFO][1][MainProcess][MainThread] - core.schains.process_manager:95 - Fetching schains to monitor... [2023-09-07 17:29:22,491 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:29:22,491 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:29:22,492 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 17:29:28,709 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:29:28,710 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:29:28,710 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:29:28,710 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:29:28,710 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:29:28,711 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:29:28,713 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:29:28,715 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 17:29:28,744 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:29:28,960 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:29:29,114 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:29:29,244 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:29:29,245 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2501 [2023-09-07 17:29:29,246 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:29:29,246 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:29:29,246 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:29:29,246 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:29:29,247 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:29:29,253 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:29:29,255 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:29:29.255096 [2023-09-07 17:29:29,422 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:29:29,449 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:29:29,476 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:29:29,477 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:29:29,484 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:29:29,484 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694107769.4475071, 'te': 1694107769.4535334, 'initial_status': True} ====================================================================================================  [2023-09-07 17:29:29,486 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:29:29.486653 [2023-09-07 17:29:29,491 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:29:29,491 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:29:30,857 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107764, schain_finish_ts: 1694108304 [2023-09-07 17:29:38,282 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:29:38,284 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:29:38,289 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:29:38,291 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:29:38,293 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:29:38,297 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 17:29:38,318 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:29:38,712 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:29:38,889 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:29:39,020 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:29:39,021 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2522 [2023-09-07 17:29:39,022 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:29:39,022 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:29:39,022 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:29:39,023 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:29:39,023 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:29:39,029 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:29:39,032 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:29:39.031988 [2023-09-07 17:29:39,186 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:29:39,215 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:29:39,246 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:29:39,249 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:29:39,261 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:29:39,262 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694107779.213753, 'te': 1694107779.220219, 'initial_status': True} ====================================================================================================  [2023-09-07 17:29:39,265 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:29:39.265591 [2023-09-07 17:29:39,270 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:29:39,271 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:29:43,160 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:29:43,161 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:29:43,164 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:29:43.164177 [2023-09-07 17:29:43,174 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:29:43,174 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:29:43,189 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:29:43,199 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694107783.1729116, 'te': 1694107783.1987567, 'initial_status': True} ====================================================================================================  [2023-09-07 17:29:43,205 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:29:43.205087 [2023-09-07 17:29:43,212 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:29:43,212 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:29:46,719 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:29:46,720 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:29:46,721 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 17:29:50,488 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:29:50,488 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:29:50,490 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:29:50.490400 [2023-09-07 17:29:50,496 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:29:50,497 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:29:50,502 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:29:50,506 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694107790.4954236, 'te': 1694107790.506225, 'initial_status': True} ====================================================================================================  [2023-09-07 17:29:50,508 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:29:50.507993 [2023-09-07 17:29:50,513 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:29:50,514 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:29:55,983 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:29:55,983 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:29:55,983 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:29:55,984 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:29:55,984 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:29:55,984 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:29:56,009 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:29:56,308 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:29:56,452 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:29:56,573 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:29:56,574 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2514 [2023-09-07 17:29:56,575 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:29:56,575 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:29:56,575 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:29:56,576 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:29:56,576 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:29:56,583 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:29:56,585 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:29:56.585077 [2023-09-07 17:29:56,773 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:29:56,801 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:29:56,830 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:29:56,833 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:29:56,840 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:29:56,840 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694107796.800039, 'te': 1694107796.8061836, 'initial_status': True} ====================================================================================================  [2023-09-07 17:29:56,843 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:29:56.843442 [2023-09-07 17:29:56,848 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:29:56,849 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:30:06,917 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107800, schain_finish_ts: 1693412520 [2023-09-07 17:30:09,663 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107800, schain_finish_ts: 0 [2023-09-07 17:30:11,681 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:30:11,681 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:30:11,684 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:30:11.684825 [2023-09-07 17:30:11,694 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:30:11,694 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:30:11,707 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:30:11,712 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694107811.6928174, 'te': 1694107811.711388, 'initial_status': True} ====================================================================================================  [2023-09-07 17:30:11,716 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:30:11.716870 [2023-09-07 17:30:11,723 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:30:11,723 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:30:13,337 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107800, schain_finish_ts: 1692637668 [2023-09-07 17:30:20,695 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107812, schain_finish_ts: 1692377364 [2023-09-07 17:30:23,984 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107812, schain_finish_ts: 1694108304 [2023-09-07 17:30:26,197 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107824, schain_finish_ts: 1693742304 [2023-09-07 17:30:29,135 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:30:29,136 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:30:29,136 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:30:29,136 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:30:29,137 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:30:29,137 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:30:29,139 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:30:29,140 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:30:29,169 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:30:29,480 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:30:29,659 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:30:29,785 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:30:29,787 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2519 [2023-09-07 17:30:29,789 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:30:29,790 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:30:29,790 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:30:29,791 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:30:29,792 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:30:29,801 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:30:29,803 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:30:29.803580 [2023-09-07 17:30:29,953 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:30:29,957 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:30:29,957 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 17:30:29,976 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:30:30,014 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:30:30,051 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:30:30,053 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:30:30,062 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:30:30,063 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694107830.0108795, 'te': 1694107830.022375, 'initial_status': True} ====================================================================================================  [2023-09-07 17:30:30,064 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:30:30.064919 [2023-09-07 17:30:30,069 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:30:30,070 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:30:34,641 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107824, schain_finish_ts: 1693742004 [2023-09-07 17:30:34,871 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:30:34,871 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:30:34,871 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:30:34,871 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:30:34,871 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:30:34,872 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:30:34,873 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:30:34,876 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:30:34,893 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:30:34,894 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:30:34,894 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:30:34,895 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:30:34,896 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:30:34,896 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:30:34,896 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:30:34,896 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:30:34,897 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:30:34,897 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:30:34,897 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:30:34,898 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:30:34,898 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:30:34,901 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:30:34,902 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 17:30:34,920 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:30:34,932 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:30:35,155 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:30:35,389 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:30:35,484 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.transactions.tools:190 - Tx complaint_bad_data completed after 1/3 retries [2023-09-07 17:30:35,485 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:354 - sChain: free-basic-rosy-SKALE. 14 node sent a complaint on 3 node [2023-09-07 17:30:35,485 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:259 - sChain: free-basic-rosy-SKALE. Will wait for FailedDkg event [2023-09-07 17:30:35,570 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:30:35,697 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:30:35,807 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:30:35,811 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2502 [2023-09-07 17:30:35,814 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:30:35,815 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:30:35,815 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:30:35,816 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:30:35,816 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:30:35,824 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:30:35,829 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:30:35.829052 [2023-09-07 17:30:35,947 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:30:36,113 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:30:36,116 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2523 [2023-09-07 17:30:36,118 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:30:36,118 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:30:36,120 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:30:36,120 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:30:36,121 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:30:36,133 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:30:36,139 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:30:36.139208 [2023-09-07 17:30:36,154 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:30:36,336 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:30:36,339 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:30:36,339 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:30:36,339 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:30:36,345 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:30:36,346 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2517 [2023-09-07 17:30:36,347 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:30:36,347 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:30:36,348 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:30:36,348 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:30:36,348 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:30:36,357 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:30:36,359 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:30:36.359804 [2023-09-07 17:30:36,379 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:30:36,419 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:30:36,427 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:30:36,438 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:30:36,439 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694107836.376773, 'te': 1694107836.3861861, 'initial_status': True} ====================================================================================================  [2023-09-07 17:30:36,442 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:30:36.442486 [2023-09-07 17:30:36,448 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:30:36,450 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:30:36,620 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:30:36,650 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:30:36,688 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:30:36,691 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:30:36,698 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:30:36,698 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694107836.6487992, 'te': 1694107836.6559894, 'initial_status': True} ====================================================================================================  [2023-09-07 17:30:36,700 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:30:36.700239 [2023-09-07 17:30:36,705 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:30:36,706 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:30:36,795 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:30:36,825 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:30:36,859 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:30:36,861 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:30:36,868 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:30:36,868 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694107836.8233893, 'te': 1694107836.8298998, 'initial_status': True} ====================================================================================================  [2023-09-07 17:30:36,870 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:30:36.870481 [2023-09-07 17:30:36,874 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:30:36,875 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:30:48,818 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:30:48,819 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:30:48,819 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:30:48,819 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:30:48,819 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:30:48,820 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:30:48,844 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:30:49,011 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:30:49,140 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:30:49,233 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:30:49,234 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2515 [2023-09-07 17:30:49,235 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:30:49,235 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:30:49,235 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:30:49,236 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:30:49,236 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:30:49,242 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:30:49,244 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:30:49.244381 [2023-09-07 17:30:49,616 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:30:49,648 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:30:49,676 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:30:49,677 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:30:49,684 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:30:49,685 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694107849.6452444, 'te': 1694107849.6530933, 'initial_status': True} ====================================================================================================  [2023-09-07 17:30:49,686 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:30:49.686680 [2023-09-07 17:30:49,690 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:30:49,691 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:30:51,439 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:30:53,181 INFO][1][MainProcess][MainThread] - core.schains.process_manager:109 - Get leaving_history for node ... [2023-09-07 17:30:53,340 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:30:53,340 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:30:53,342 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:30:53.342798 [2023-09-07 17:30:53,351 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:30:53,351 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:30:53,358 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:30:53,369 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694107853.3498673, 'te': 1694107853.369253, 'initial_status': True} ====================================================================================================  [2023-09-07 17:30:53,372 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:30:53.372941 [2023-09-07 17:30:53,387 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:30:53,387 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:30:56,501 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:30:56,501 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:30:56,505 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:30:56.505326 [2023-09-07 17:30:56,514 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:30:56,514 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:30:56,520 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:30:56,524 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694107856.511525, 'te': 1694107856.5245256, 'initial_status': True} ====================================================================================================  [2023-09-07 17:30:56,526 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:30:56.526961 [2023-09-07 17:30:56,532 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:30:56,532 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:31:02,847 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:31:02,847 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:31:02,851 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:31:02.851401 [2023-09-07 17:31:02,859 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:31:02,859 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:31:02,865 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:31:02,871 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694107862.8580227, 'te': 1694107862.8710217, 'initial_status': True} ====================================================================================================  [2023-09-07 17:31:02,873 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:31:02.873160 [2023-09-07 17:31:02,877 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:31:02,877 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:31:04,176 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107860, schain_finish_ts: 1693412520 [2023-09-07 17:31:19,232 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107872, schain_finish_ts: 1692637668 [2023-09-07 17:31:23,118 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107872, schain_finish_ts: 1692377616 [2023-09-07 17:31:23,794 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107872, schain_finish_ts: 1692377364 [2023-09-07 17:31:25,867 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107872, schain_finish_ts: 1694108304 [2023-09-07 17:31:28,629 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:31:29,705 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:31:29,706 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:31:29,706 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:31:29,706 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:31:29,706 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:31:29,707 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:31:29,707 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:31:29,713 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 17:31:29,753 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:31:30,042 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:31:30,059 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107884, schain_finish_ts: 1693742304 [2023-09-07 17:31:30,246 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:31:30,368 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:31:30,369 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2520 [2023-09-07 17:31:30,370 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:31:30,370 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:31:30,370 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:31:30,371 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:31:30,371 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:31:30,380 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:31:30,386 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:31:30.386250 [2023-09-07 17:31:30,700 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:31:30,731 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:31:30,765 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:31:30,767 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:31:30,776 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:31:30,777 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694107890.7284827, 'te': 1694107890.7380295, 'initial_status': True} ====================================================================================================  [2023-09-07 17:31:30,779 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:31:30.779378 [2023-09-07 17:31:30,785 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:31:30,785 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:31:31,953 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107884, schain_finish_ts: 0 [2023-09-07 17:31:37,572 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107884, schain_finish_ts: 1693742004 [2023-09-07 17:31:47,602 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:31:47,604 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:31:47,604 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:31:47,605 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:31:47,605 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:31:47,606 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 17:31:47,618 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:31:47,619 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:31:47,619 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:31:47,619 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:31:47,620 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:31:47,620 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:31:47,622 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:31:47,622 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:31:47,639 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:31:47,658 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:31:47,954 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:31:48,101 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:31:48,239 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:31:48,353 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:31:48,355 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2524 [2023-09-07 17:31:48,357 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:31:48,357 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:31:48,358 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:31:48,358 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:31:48,359 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:31:48,369 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:31:48,372 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:31:48.372789 [2023-09-07 17:31:48,466 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:31:48,580 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:31:48,581 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2518 [2023-09-07 17:31:48,581 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:31:48,582 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:31:48,582 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:31:48,582 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:31:48,583 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:31:48,588 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:31:48,590 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:31:48.590885 [2023-09-07 17:31:48,737 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:31:48,777 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:31:48,823 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:31:48,825 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:31:48,835 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:31:48,836 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694107908.772911, 'te': 1694107908.7843697, 'initial_status': True} ====================================================================================================  [2023-09-07 17:31:48,839 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:31:48.839239 [2023-09-07 17:31:48,843 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:31:48,844 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:31:48,972 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:31:48,997 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:31:49,026 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:31:49,028 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:31:49,038 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:31:49,039 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694107908.9961243, 'te': 1694107909.002608, 'initial_status': True} ====================================================================================================  [2023-09-07 17:31:49,044 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:31:49.044451 [2023-09-07 17:31:49,051 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:31:49,052 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:31:51,205 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:31:51,206 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:31:51,206 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:31:51,207 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:31:51,207 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:31:51,208 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:31:51,211 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:31:51,212 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:31:51,212 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:31:51,214 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:31:51.214316 [2023-09-07 17:31:51,216 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:31:51,226 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:31:51,227 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:31:51,235 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:31:51,241 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694107911.2213407, 'te': 1694107911.2409513, 'initial_status': True} ====================================================================================================  [2023-09-07 17:31:51,246 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:31:51.246148 [2023-09-07 17:31:51,247 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:31:51,251 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:31:51,252 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:31:51,258 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107884, schain_finish_ts: 0 [2023-09-07 17:31:51,445 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:31:51,591 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:31:51,712 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:31:51,713 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2503 [2023-09-07 17:31:51,714 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:31:51,714 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:31:51,714 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:31:51,715 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:31:51,715 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:31:51,722 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:31:51,724 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:31:51.724696 [2023-09-07 17:31:51,887 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:31:51,917 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:31:51,949 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:31:51,951 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:31:51,958 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:31:51,959 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694107911.9146962, 'te': 1694107911.9229023, 'initial_status': True} ====================================================================================================  [2023-09-07 17:31:51,960 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:31:51.960825 [2023-09-07 17:31:51,965 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:31:51,966 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:31:54,365 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:31:54,367 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:31:54,367 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:31:57,351 INFO][1][MainProcess][MainThread] - core.schains.process_manager:117 - Got leaving sChains for the node: [] [2023-09-07 17:31:57,351 INFO][1][MainProcess][MainThread] - core.schains.process_manager:101 -  ==================================================================================================== Monitoring sChains Node ID: 16 sChains on node: [{'name': 'portly-passionate-sirius', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666281384, 'startBlock': 7803380, 'deposit': 0, 'index': 1, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x0000017723ca2', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'curly-red-alterf', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1676058936, 'startBlock': 8469646, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000001fe6cdfa', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'howling-hoedus', 'mainnetOwner': '0x1026467C91fdA40aCc7e6935d1E52f67274FD669', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666181400, 'startBlock': 7796660, 'deposit': 0, 'index': 0, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0xe5a600f4c2608', 'multitransactionMode': False, 'thresholdEncryption': False, 'active': True}, {'name': 'green-giddy-denebola', 'mainnetOwner': '0x8e835d5d9E28ffD259E8Fa8988Cacfc333A582B4', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674565164, 'startBlock': 8368049, 'deposit': 0, 'index': 4, 'generation': 1, 'originator': '0xd2001DAb6898127Be2F167B548691C87251D13C3', 'chainId': '0x00000585eb4b1', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'wan-tasty-kitalpha', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674568392, 'startBlock': 8368271, 'deposit': 0, 'index': 5, 'generation': 1, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x00000152e7be9', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'free-basic-rosy-SKALE', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1693996920, 'startBlock': 9646443, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000002bb90439', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'submissive-well-off-rotanev', 'mainnetOwner': '0x487360568fE5Ee4AE5c75782D5F63e9935B0E512', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1692361104, 'startBlock': 9539053, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000003791fa10', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}] Number of sChains on node: 7 Empty sChain structs: 0 ====================================================================================================  [2023-09-07 17:32:02,528 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:32:02,528 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain portly-passionate-sirius - Process is running: PID = 7 [2023-09-07 17:32:04,304 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:32:05,989 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:32:05,991 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:32:05,994 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:32:06,282 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:32:06,283 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:32:06,283 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:32:06,283 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:32:06,283 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:32:06,284 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:32:06,311 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:32:06,485 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:32:06,639 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:32:06,743 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:32:06,744 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2516 [2023-09-07 17:32:06,745 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:32:06,745 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:32:06,745 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:32:06,746 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:32:06,746 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:32:06,753 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:32:06,755 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:32:06.755181 [2023-09-07 17:32:06,894 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:32:06,922 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:32:06,970 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:32:06,972 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:32:06,980 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:32:06,981 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694107926.920767, 'te': 1694107926.927216, 'initial_status': True} ====================================================================================================  [2023-09-07 17:32:06,987 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:32:06.987894 [2023-09-07 17:32:06,995 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:32:06,996 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:32:10,947 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:32:10,947 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain curly-red-alterf - Process is running: PID = 8 [2023-09-07 17:32:13,871 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:32:13,871 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain howling-hoedus - Process is running: PID = 9 [2023-09-07 17:32:15,545 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107932, schain_finish_ts: 1693742304 [2023-09-07 17:32:16,927 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:32:16,927 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:32:16,929 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:32:16.929752 [2023-09-07 17:32:16,938 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:32:16,939 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:32:16,945 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:32:16,949 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694107936.9371374, 'te': 1694107936.9492087, 'initial_status': True} ====================================================================================================  [2023-09-07 17:32:16,951 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:32:16.951155 [2023-09-07 17:32:16,955 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:32:16,955 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:32:18,115 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107932, schain_finish_ts: 1693412520 [2023-09-07 17:32:18,454 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:32:18,454 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain green-giddy-denebola - Process is running: PID = 10 [2023-09-07 17:32:26,701 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:32:26,702 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain wan-tasty-kitalpha - Process is running: PID = 11 [2023-09-07 17:32:27,681 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107932, schain_finish_ts: 1692637668 [2023-09-07 17:32:29,573 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:32:29,573 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:32:29,575 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:32:29.575802 [2023-09-07 17:32:29,582 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:32:29,582 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:32:29,588 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:32:29,592 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694107949.5809398, 'te': 1694107949.5921638, 'initial_status': True} ====================================================================================================  [2023-09-07 17:32:29,594 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:32:29.594198 [2023-09-07 17:32:29,598 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:32:29,599 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:32:29,648 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107932, schain_finish_ts: 1692377364 [2023-09-07 17:32:30,580 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:32:30,580 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain free-basic-rosy-SKALE - Process is running: PID = 30622 [2023-09-07 17:32:33,715 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:32:33,715 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:32:33,716 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:32:33,716 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:32:33,716 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:32:33,716 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:32:33,718 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:32:33,722 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:32:33,743 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:32:34,039 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:32:34,218 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:32:34,320 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:32:34,321 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2521 [2023-09-07 17:32:34,322 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:32:34,322 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:32:34,322 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:32:34,323 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:32:34,323 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:32:34,329 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:32:34,331 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:32:34.331146 [2023-09-07 17:32:34,483 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:32:34,515 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:32:34,548 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:32:34,551 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:32:34,558 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:32:34,559 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694107954.5131981, 'te': 1694107954.5195253, 'initial_status': True} ====================================================================================================  [2023-09-07 17:32:34,560 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:32:34.560823 [2023-09-07 17:32:34,565 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:32:34,566 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:32:35,624 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:32:35,624 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:32:35,625 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:32:35,625 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:32:35,625 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:32:35,625 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:32:35,627 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:32:35,628 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:32:35,660 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:32:35,672 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:32:35,672 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:32:35,672 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:32:35,672 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:32:35,672 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:32:35,672 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:32:35,694 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:32:35,856 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:32:35,983 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:32:36,149 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:32:36,269 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:32:36,270 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2519 [2023-09-07 17:32:36,271 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:32:36,271 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:32:36,272 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:32:36,272 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:32:36,272 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:32:36,279 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:32:36,281 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:32:36.281088 [2023-09-07 17:32:36,298 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:32:36,407 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:32:36,408 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2525 [2023-09-07 17:32:36,409 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:32:36,409 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:32:36,409 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:32:36,410 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:32:36,410 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:32:36,416 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:32:36,418 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:32:36.418663 [2023-09-07 17:32:36,436 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:32:36,460 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:32:36,495 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:32:36,497 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:32:36,503 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:32:36,506 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694107956.459219, 'te': 1694107956.465146, 'initial_status': True} ====================================================================================================  [2023-09-07 17:32:36,556 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:32:36.556826 [2023-09-07 17:32:36,573 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:32:36,580 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:32:36,581 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:32:36,601 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:32:36,635 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:32:36,637 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:32:36,646 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:32:36,647 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694107956.600109, 'te': 1694107956.6055465, 'initial_status': True} ====================================================================================================  [2023-09-07 17:32:36,649 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:32:36.649044 [2023-09-07 17:32:36,654 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:32:36,658 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:32:37,117 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:32:37,120 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain submissive-well-off-rotanev - Process is running: PID = 24912 [2023-09-07 17:32:37,120 INFO][1][MainProcess][MainThread] - core.schains.process_manager:88 - Process manager procedure finished [2023-09-07 17:32:37,120 INFO][1][MainProcess][MainThread] - __main__:65 - Sleeping for 90s after run_process_manager [2023-09-07 17:32:39,055 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:32:39,056 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:32:39,056 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:32:39,056 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:32:39,056 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:32:39,057 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:32:39,059 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:32:39,062 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:32:39,110 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:32:39,325 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:32:39,454 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:32:39,558 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:32:39,559 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2504 [2023-09-07 17:32:39,560 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:32:39,560 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:32:39,560 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:32:39,560 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:32:39,561 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:32:39,567 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:32:39,568 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:32:39.568920 [2023-09-07 17:32:39,799 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:32:39,827 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:32:39,859 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:32:39,861 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:32:39,870 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:32:39,870 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694107959.826096, 'te': 1694107959.831904, 'initial_status': True} ====================================================================================================  [2023-09-07 17:32:39,872 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:32:39.872163 [2023-09-07 17:32:39,880 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:32:39,881 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:32:40,960 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107956, schain_finish_ts: 0 [2023-09-07 17:32:46,647 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:32:46,647 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:32:46,649 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:32:46.649479 [2023-09-07 17:32:46,655 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:32:46,656 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:32:46,661 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:32:46,665 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694107966.654417, 'te': 1694107966.665688, 'initial_status': True} ====================================================================================================  [2023-09-07 17:32:46,667 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:32:46.667400 [2023-09-07 17:32:46,671 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:32:46,671 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:32:50,313 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:32:50,314 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:32:50,319 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:32:50.319100 [2023-09-07 17:32:50,328 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:32:50,328 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:32:50,333 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:32:50,339 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694107970.3266015, 'te': 1694107970.339394, 'initial_status': True} ====================================================================================================  [2023-09-07 17:32:50,341 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:32:50.341484 [2023-09-07 17:32:50,345 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:32:50,345 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:32:52,940 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107968, schain_finish_ts: 1694108304 [2023-09-07 17:32:54,346 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107968, schain_finish_ts: 1693742004 [2023-09-07 17:32:55,381 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:32:59,367 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:32:59,368 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:32:59,368 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:33:02,832 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:33:02,832 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:33:02,834 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:33:02.834798 [2023-09-07 17:33:02,841 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:33:02,842 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:33:02,848 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:33:02,853 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694107982.840621, 'te': 1694107982.8530774, 'initial_status': True} ====================================================================================================  [2023-09-07 17:33:02,855 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:33:02.855260 [2023-09-07 17:33:02,859 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:33:02,859 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:33:08,496 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:33:08,496 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:33:08,496 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:33:11,685 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107980, schain_finish_ts: 1692637668 [2023-09-07 17:33:15,533 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:33:15,533 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:33:15,533 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:33:15,534 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:33:15,534 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:33:15,534 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 17:33:15,568 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:33:15,776 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:33:15,942 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:33:16,056 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:33:16,057 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2517 [2023-09-07 17:33:16,058 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:33:16,058 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:33:16,058 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:33:16,059 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:33:16,059 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:33:16,066 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:33:16,068 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:33:16.068811 [2023-09-07 17:33:16,335 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:33:16,364 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:33:16,395 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:33:16,397 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:33:16,405 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:33:16,405 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694107996.3620393, 'te': 1694107996.368915, 'initial_status': True} ====================================================================================================  [2023-09-07 17:33:16,408 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:33:16.408253 [2023-09-07 17:33:16,412 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:33:16,413 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:33:17,476 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107980, schain_finish_ts: 1692377364 [2023-09-07 17:33:22,880 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 17:33:23,045 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:33:23,046 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:33:23,046 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:33:23,047 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:33:23,211 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:33:23,211 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:33:23,214 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 17:33:23,218 ERROR][11][wan-tasty-kitalpha][T_1] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:33:23,218 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:33:24,080 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107980, schain_finish_ts: 1693412520 [2023-09-07 17:33:24,216 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107980, schain_finish_ts: 1693742304 [2023-09-07 17:33:24,246 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:33:24,246 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:33:24,246 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:33:24,246 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:33:24,246 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:33:24,247 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:33:24,247 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:33:24,249 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:33:24,281 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:33:24,468 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:33:24,592 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:33:24,690 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:33:24,691 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2520 [2023-09-07 17:33:24,692 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:33:24,692 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:33:24,692 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:33:24,693 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:33:24,693 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:33:24,699 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:33:24,702 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:33:24.702286 [2023-09-07 17:33:24,973 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:33:25,008 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:33:25,076 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:33:25,079 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:33:25,086 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:33:25,087 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694108005.0071363, 'te': 1694108005.0177023, 'initial_status': True} ====================================================================================================  [2023-09-07 17:33:25,088 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:33:25.088846 [2023-09-07 17:33:25,092 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:33:25,093 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:33:25,123 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:33:25,123 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:33:25,124 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:33:25,124 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:33:25,125 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:33:25,125 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:33:25,127 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:33:25,127 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:33:25,160 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:33:25,341 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:33:25,477 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:33:25,582 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:33:25,584 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2505 [2023-09-07 17:33:25,585 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:33:25,585 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:33:25,586 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:33:25,586 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:33:25,587 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:33:25,594 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:33:25,597 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:33:25.597687 [2023-09-07 17:33:25,767 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:33:25,798 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:33:25,828 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:33:25,830 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:33:25,838 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:33:25,838 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694108005.7961016, 'te': 1694108005.803626, 'initial_status': True} ====================================================================================================  [2023-09-07 17:33:25,841 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:33:25.841398 [2023-09-07 17:33:25,845 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:33:25,846 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:33:33,361 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:33:34,479 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:33:34,479 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:33:34,479 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:33:34,480 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 17:33:34,480 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 17:33:34,480 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:33:34,487 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:33:34,492 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 17:33:34,520 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:33:34,847 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:33:34,995 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:33:35,103 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:33:35,104 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2526 [2023-09-07 17:33:35,105 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:33:35,105 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:33:35,106 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:33:35,106 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:33:35,106 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:33:35,112 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:33:35,114 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:33:35.114741 [2023-09-07 17:33:35,246 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:33:35,276 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:33:35,305 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:33:35,306 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:33:35,313 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:33:35,314 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694108015.27445, 'te': 1694108015.2809632, 'initial_status': True} ====================================================================================================  [2023-09-07 17:33:35,316 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:33:35.316052 [2023-09-07 17:33:35,323 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:33:35,323 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:33:38,806 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:33:38,807 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:33:38,807 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:33:38,808 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:33:38,809 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:33:38,809 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:33:38,816 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:33:38,817 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 17:33:38,846 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:33:39,056 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:33:39,301 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:33:39,440 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:33:39,441 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2522 [2023-09-07 17:33:39,441 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:33:39,442 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:33:39,442 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:33:39,442 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:33:39,443 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:33:39,449 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:33:39,451 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:33:39.451168 [2023-09-07 17:33:39,768 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:33:39,792 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:33:39,818 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:33:39,820 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:33:39,827 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:33:39,827 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694108019.7910366, 'te': 1694108019.7969644, 'initial_status': True} ====================================================================================================  [2023-09-07 17:33:39,829 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:33:39.829019 [2023-09-07 17:33:39,833 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:33:39,834 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:33:41,805 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107980, schain_finish_ts: 0 [2023-09-07 17:33:44,520 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694107980, schain_finish_ts: 1694108304 [2023-09-07 17:33:54,094 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:33:54,094 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:33:54,097 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:33:54.097337 [2023-09-07 17:33:54,104 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:33:54,104 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:33:54,109 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:33:54,114 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694108034.1027431, 'te': 1694108034.114036, 'initial_status': True} ====================================================================================================  [2023-09-07 17:33:54,115 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:33:54.115889 [2023-09-07 17:33:54,120 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:33:54,120 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:34:00,472 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 17:34:00,472 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 17:34:00,565 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108028, schain_finish_ts: 1692637668 [2023-09-07 17:34:05,164 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:34:05,164 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:34:05,167 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:34:05.167827 [2023-09-07 17:34:05,174 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:34:05,175 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:34:05,180 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:34:05,185 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694108045.173405, 'te': 1694108045.1849005, 'initial_status': True} ====================================================================================================  [2023-09-07 17:34:05,186 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:34:05.186749 [2023-09-07 17:34:05,191 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:34:05,191 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:34:05,269 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:34:05,270 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:34:05,270 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 17:34:06,463 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108040, schain_finish_ts: 1693742004 [2023-09-07 17:34:07,213 INFO][1][MainProcess][MainThread] - core.schains.cleaner:65 - Cleaner process started [2023-09-07 17:34:07,246 INFO][30765][cleaner][MainThread] - core.schains.cleaner:110 - Cleaner procedure started. [2023-09-07 17:34:16,203 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:34:16,203 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:34:16,204 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 17:34:17,728 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108040, schain_finish_ts: 1692377364 [2023-09-07 17:34:22,021 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108040, schain_finish_ts: 1693412520 [2023-09-07 17:34:22,212 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:34:22,212 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:34:22,213 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:34:22,213 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:34:22,213 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:34:22,214 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:34:22,217 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:34:22,218 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108040, schain_finish_ts: 1693742304 [2023-09-07 17:34:22,218 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:34:22,241 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:34:22,478 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:34:22,616 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:34:22,714 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:34:22,715 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2521 [2023-09-07 17:34:22,716 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:34:22,716 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:34:22,717 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:34:22,717 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:34:22,718 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:34:22,726 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:34:22,728 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:34:22.728760 [2023-09-07 17:34:22,962 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:34:23,000 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:34:23,033 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:34:23,034 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:34:23,041 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:34:23,042 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694108062.9960306, 'te': 1694108063.0073316, 'initial_status': True} ====================================================================================================  [2023-09-07 17:34:23,043 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:34:23.043951 [2023-09-07 17:34:23,048 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:34:23,049 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:34:23,703 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:34:23,705 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:34:23,708 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:34:23,712 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:34:23,714 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:34:23,714 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 17:34:23,718 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:34:23,718 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:34:23,721 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:34:23.721186 [2023-09-07 17:34:23,728 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:34:23,728 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:34:23,735 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:34:23,740 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694108063.7269812, 'te': 1694108063.7400038, 'initial_status': True} ====================================================================================================  [2023-09-07 17:34:23,740 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:34:23,743 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:34:23.743333 [2023-09-07 17:34:23,748 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:34:23,748 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:34:23,839 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:34:23,839 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:34:23,842 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:34:23.842039 [2023-09-07 17:34:23,847 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:34:24,002 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:34:24,182 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:34:24,308 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:34:24,309 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2518 [2023-09-07 17:34:24,310 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:34:24,310 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:34:24,310 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:34:24,311 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:34:24,311 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:34:24,317 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:34:24,319 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:34:24.319421 [2023-09-07 17:34:24,456 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:34:24,480 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:34:24,508 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:34:24,510 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:34:24,517 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:34:24,517 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694108064.4791422, 'te': 1694108064.4854271, 'initial_status': True} ====================================================================================================  [2023-09-07 17:34:24,519 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:34:24.519146 [2023-09-07 17:34:24,523 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:34:24,524 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:34:31,577 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:34:32,809 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:34:32,809 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:34:32,809 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:34:32,810 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:34:32,810 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:34:32,810 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:34:32,812 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:34:32,815 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:34:32,827 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:34:32,827 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:34:32,827 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:34:32,828 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:34:32,828 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:34:32,828 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:34:32,829 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:34:32,835 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 17:34:32,838 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:34:32,854 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:34:33,030 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:34:33,230 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:34:33,386 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:34:33,486 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:34:33,487 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2523 [2023-09-07 17:34:33,488 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:34:33,490 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:34:33,490 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:34:33,490 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:34:33,491 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:34:33,497 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:34:33,499 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:34:33.499578 [2023-09-07 17:34:33,608 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:34:33,710 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:34:33,711 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2506 [2023-09-07 17:34:33,711 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:34:33,712 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:34:33,712 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:34:33,712 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:34:33,712 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:34:33,719 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:34:33,721 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:34:33.721041 [2023-09-07 17:34:33,734 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:34:33,770 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:34:33,798 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:34:33,800 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:34:33,807 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:34:33,807 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694108073.768429, 'te': 1694108073.775095, 'initial_status': True} ====================================================================================================  [2023-09-07 17:34:33,808 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:34:33.808937 [2023-09-07 17:34:33,813 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:34:33,814 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:34:33,857 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:34:33,883 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:34:33,913 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:34:33,917 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:34:33,928 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:34:33,929 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694108073.8811831, 'te': 1694108073.8883607, 'initial_status': True} ====================================================================================================  [2023-09-07 17:34:33,932 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:34:33.932334 [2023-09-07 17:34:33,938 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:34:33,939 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:34:35,076 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:34:35,076 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:34:35,076 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:34:35,077 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:34:35,077 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:34:35,077 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:34:35,100 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:34:35,308 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:34:35,460 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:34:35,563 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:34:35,564 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2527 [2023-09-07 17:34:35,565 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:34:35,565 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:34:35,566 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:34:35,566 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:34:35,566 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:34:35,572 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:34:35,574 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:34:35.574785 [2023-09-07 17:34:35,808 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:34:35,835 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:34:35,868 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:34:35,870 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:34:35,878 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:34:35,878 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694108075.8342912, 'te': 1694108075.8405027, 'initial_status': True} ====================================================================================================  [2023-09-07 17:34:35,880 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:34:35.880534 [2023-09-07 17:34:35,885 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:34:35,885 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:34:50,148 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108076, schain_finish_ts: 1694108304 [2023-09-07 17:34:50,519 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108088, schain_finish_ts: 0 [2023-09-07 17:34:51,384 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108088, schain_finish_ts: 1693742004 [2023-09-07 17:34:59,130 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:34:59,131 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:34:59,134 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:34:59.134814 [2023-09-07 17:34:59,141 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:34:59,142 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:34:59,148 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:34:59,154 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694108099.1404083, 'te': 1694108099.1539657, 'initial_status': True} ====================================================================================================  [2023-09-07 17:34:59,155 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:34:59.155953 [2023-09-07 17:34:59,163 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:34:59,163 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:35:00,571 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:35:00,571 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:35:00,573 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:35:00.573288 [2023-09-07 17:35:00,581 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:35:00,581 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:35:00,587 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:35:00,591 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694108100.579428, 'te': 1694108100.5915058, 'initial_status': True} ====================================================================================================  [2023-09-07 17:35:00,593 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:35:00.593461 [2023-09-07 17:35:00,597 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:35:00,598 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:35:02,349 INFO][30765][cleaner][MainThread] - core.schains.cleaner:116 - sChains on contracts: ['portly-passionate-sirius', 'curly-red-alterf', 'howling-hoedus', 'green-giddy-denebola', 'wan-tasty-kitalpha', 'free-basic-rosy-SKALE', 'submissive-well-off-rotanev'] sChains on node: ['curly-red-alterf', 'free-basic-rosy-SKALE', 'green-giddy-denebola', 'howling-hoedus', 'portly-passionate-sirius', 'submissive-well-off-rotanev', 'wan-tasty-kitalpha'] [2023-09-07 17:35:02,350 INFO][30765][cleaner][MainThread] - core.schains.cleaner:132 - Cleanup procedure finished [2023-09-07 17:35:02,355 INFO][1][MainProcess][MainThread] - core.schains.cleaner:67 - Cleaner process is joined. [2023-09-07 17:35:02,356 INFO][1][MainProcess][MainThread] - core.schains.cleaner:68 - Terminating the process [2023-09-07 17:35:02,356 INFO][1][MainProcess][MainThread] - __main__:70 - Sleeping for 90s after run_cleaner [2023-09-07 17:35:06,753 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:35:06,753 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:35:06,754 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:35:07,978 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:35:07,978 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:35:07,978 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:35:07,979 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:35:07,979 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:35:07,980 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 17:35:08,000 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:35:08,213 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:35:08,358 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:35:08,473 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:35:08,474 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2519 [2023-09-07 17:35:08,475 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:35:08,475 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:35:08,476 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:35:08,476 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:35:08,477 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:35:08,484 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:35:08,487 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:35:08.487683 [2023-09-07 17:35:08,712 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108100, schain_finish_ts: 1692377364 [2023-09-07 17:35:08,765 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:35:08,793 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:35:08,825 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:35:08,827 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:35:08,834 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:35:08,835 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694108108.7915797, 'te': 1694108108.797596, 'initial_status': True} ====================================================================================================  [2023-09-07 17:35:08,837 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:35:08.837490 [2023-09-07 17:35:08,842 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:35:08,842 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:35:10,699 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:35:10,699 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:35:10,700 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:35:13,743 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:35:13,743 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:35:13,748 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:35:13.748490 [2023-09-07 17:35:13,770 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:35:13,773 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:35:13,787 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:35:13,797 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694108113.764267, 'te': 1694108113.7961187, 'initial_status': True} ====================================================================================================  [2023-09-07 17:35:13,799 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:35:13.799526 [2023-09-07 17:35:13,805 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:35:13,805 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:35:17,149 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108112, schain_finish_ts: 1692637668 [2023-09-07 17:35:22,650 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108112, schain_finish_ts: 1693742304 [2023-09-07 17:35:22,955 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:35:22,956 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:35:22,956 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:35:22,957 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:35:22,957 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:35:22,958 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:35:22,964 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:35:22,968 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:35:22,991 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:35:23,275 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:35:23,460 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:35:23,583 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:35:23,585 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2507 [2023-09-07 17:35:23,585 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:35:23,586 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:35:23,586 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:35:23,586 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:35:23,587 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:35:23,595 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:35:23,597 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:35:23.597322 [2023-09-07 17:35:23,819 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:35:23,850 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:35:23,880 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:35:23,883 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:35:23,890 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:35:23,891 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694108123.8490322, 'te': 1694108123.8546333, 'initial_status': True} ====================================================================================================  [2023-09-07 17:35:23,892 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:35:23.892708 [2023-09-07 17:35:23,896 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:35:23,897 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:35:25,772 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:35:31,438 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108124, schain_finish_ts: 1693412520 [2023-09-07 17:35:39,089 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:35:39,089 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:35:39,089 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:35:39,090 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:35:39,090 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:35:39,090 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:35:39,092 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:35:39,093 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:35:39,120 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:35:39,308 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:35:39,539 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:35:39,634 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:35:39,636 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2522 [2023-09-07 17:35:39,636 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:35:39,637 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:35:39,637 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:35:39,637 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:35:39,637 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:35:39,643 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:35:39,645 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:35:39.645513 [2023-09-07 17:35:39,785 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:35:39,809 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:35:39,837 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:35:39,839 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:35:39,846 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:35:39,846 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694108139.807415, 'te': 1694108139.8134577, 'initial_status': True} ====================================================================================================  [2023-09-07 17:35:39,851 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:35:39.851969 [2023-09-07 17:35:39,858 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:35:39,862 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:35:59,101 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:35:59,104 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:35:59,104 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:35:59,105 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:35:59,106 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:35:59,107 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:35:59,141 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:35:59,345 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:35:59,489 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:35:59,580 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:35:59,581 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2528 [2023-09-07 17:35:59,582 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:35:59,586 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:35:59,587 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:35:59,587 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:35:59,587 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:35:59,593 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:35:59,595 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:35:59.595822 [2023-09-07 17:35:59,728 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:35:59,752 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:35:59,778 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:35:59,780 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:35:59,787 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:35:59,787 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694108159.750519, 'te': 1694108159.756594, 'initial_status': True} ====================================================================================================  [2023-09-07 17:35:59,789 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:35:59.789434 [2023-09-07 17:35:59,794 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:35:59,794 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:36:01,178 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108148, schain_finish_ts: 0 [2023-09-07 17:36:01,819 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:36:01,819 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:36:01,821 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:36:01.821444 [2023-09-07 17:36:01,829 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:36:01,829 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:36:01,836 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:36:01,842 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694108161.8282888, 'te': 1694108161.842423, 'initial_status': True} ====================================================================================================  [2023-09-07 17:36:01,844 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:36:01.844508 [2023-09-07 17:36:01,849 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:36:01,849 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:36:04,309 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108148, schain_finish_ts: 1694108304 [2023-09-07 17:36:06,378 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108148, schain_finish_ts: 1692377364 [2023-09-07 17:36:06,587 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:36:06,587 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:36:06,588 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:36:06,588 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:36:06,589 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:36:06,589 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:36:06,592 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:36:06,595 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:36:06,625 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:36:06,844 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:36:07,150 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:36:07,255 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:36:07,256 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2524 [2023-09-07 17:36:07,257 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:36:07,257 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:36:07,257 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:36:07,258 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:36:07,258 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:36:07,265 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:36:07,268 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:36:07.268394 [2023-09-07 17:36:07,415 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:36:07,442 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:36:07,471 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:36:07,473 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:36:07,481 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:36:07,482 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694108167.441148, 'te': 1694108167.4469998, 'initial_status': True} ====================================================================================================  [2023-09-07 17:36:07,484 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:36:07.484657 [2023-09-07 17:36:07,489 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:36:07,489 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:36:08,204 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108148, schain_finish_ts: 1693742004 [2023-09-07 17:36:12,648 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:36:12,752 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:36:12,753 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:36:12,754 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:36:12.754780 [2023-09-07 17:36:12,761 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:36:12,761 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:36:12,767 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:36:12,772 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694108172.7597926, 'te': 1694108172.7723367, 'initial_status': True} ====================================================================================================  [2023-09-07 17:36:12,774 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:36:12.774302 [2023-09-07 17:36:12,779 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:36:12,779 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:36:17,878 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:36:17,878 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:36:17,878 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 17:36:18,160 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108172, schain_finish_ts: 1692637668 [2023-09-07 17:36:18,357 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:36:18,358 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:36:18,358 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:36:18,358 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:36:18,358 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:36:18,358 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:36:18,360 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:36:18,363 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:36:18,398 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:36:18,644 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:36:18,905 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:36:19,005 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:36:19,006 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2508 [2023-09-07 17:36:19,007 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:36:19,007 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:36:19,007 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:36:19,008 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:36:19,008 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:36:19,015 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:36:19,017 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:36:19.017209 [2023-09-07 17:36:19,168 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:36:19,196 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:36:19,236 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:36:19,239 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:36:19,248 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:36:19,248 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694108179.1946905, 'te': 1694108179.201752, 'initial_status': True} ====================================================================================================  [2023-09-07 17:36:19,251 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:36:19.251204 [2023-09-07 17:36:19,256 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:36:19,256 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:36:21,506 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:36:21,506 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:36:21,506 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:36:21,507 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:36:21,507 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:36:21,507 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:36:21,509 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:36:21,511 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:36:21,532 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:36:21,701 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:36:21,837 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:36:21,933 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:36:21,934 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2523 [2023-09-07 17:36:21,935 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:36:21,935 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:36:21,935 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:36:21,936 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:36:21,936 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:36:21,941 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:36:21,943 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:36:21.943585 [2023-09-07 17:36:22,087 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:36:22,113 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:36:22,139 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:36:22,141 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:36:22,149 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:36:22,149 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694108182.111638, 'te': 1694108182.1176383, 'initial_status': True} ====================================================================================================  [2023-09-07 17:36:22,151 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:36:22.151415 [2023-09-07 17:36:22,156 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:36:22,159 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:36:24,558 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:36:24,558 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:36:24,558 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 17:36:28,199 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:36:28,199 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:36:28,200 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:36:28,200 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:36:28,200 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:36:28,201 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 17:36:28,221 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:36:28,473 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:36:28,656 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:36:28,780 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:36:28,781 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2520 [2023-09-07 17:36:28,782 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:36:28,783 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:36:28,783 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:36:28,783 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:36:28,784 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:36:28,791 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:36:28,793 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:36:28.793539 [2023-09-07 17:36:28,954 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:36:28,982 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:36:29,014 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:36:29,017 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:36:29,025 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:36:29,025 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694108188.9803832, 'te': 1694108188.9871056, 'initial_status': True} ====================================================================================================  [2023-09-07 17:36:29,027 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:36:29.027780 [2023-09-07 17:36:29,032 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:36:29,032 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:36:32,446 INFO][1][MainProcess][MainThread] - core.schains.process_manager:57 - Process manager started [2023-09-07 17:36:36,006 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108172, schain_finish_ts: 1693742304 [2023-09-07 17:36:38,113 INFO][1][MainProcess][MainThread] - core.schains.notifications:35 - Node account has 406903463189434189 WEI [2023-09-07 17:36:38,115 INFO][1][MainProcess][MainThread] - tools.notifications.messages:188 - Saving new balance state 3013 1 [2023-09-07 17:36:38,116 INFO][1][MainProcess][MainThread] - core.schains.process_manager:95 - Fetching schains to monitor... [2023-09-07 17:36:43,857 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:36:43,858 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:36:43,861 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:36:43.861461 [2023-09-07 17:36:43,871 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:36:43,871 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:36:43,878 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:36:43,883 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694108203.8696177, 'te': 1694108203.8830578, 'initial_status': True} ====================================================================================================  [2023-09-07 17:36:43,884 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:36:43.884816 [2023-09-07 17:36:43,889 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:36:43,889 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:36:51,071 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:36:51,072 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:36:51,073 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:36:51.073975 [2023-09-07 17:36:51,081 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:36:51,081 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:36:51,087 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:36:51,092 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694108211.079611, 'te': 1694108211.0924084, 'initial_status': True} ====================================================================================================  [2023-09-07 17:36:51,094 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:36:51.094442 [2023-09-07 17:36:51,098 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:36:51,098 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:36:51,113 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108208, schain_finish_ts: 1693412520 [2023-09-07 17:36:51,694 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:36:51,703 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:36:51,703 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:36:51,705 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:36:51.705636 [2023-09-07 17:36:51,713 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:36:51,713 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:36:51,718 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:36:51,724 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694108211.7115808, 'te': 1694108211.724316, 'initial_status': True} ====================================================================================================  [2023-09-07 17:36:51,726 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:36:51.726553 [2023-09-07 17:36:51,731 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:36:51,731 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:37:00,765 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:37:00,765 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:37:00,765 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:37:00,765 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:37:00,766 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:37:00,766 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:37:00,767 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:37:00,768 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:37:00,792 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:37:01,055 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:37:01,224 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:37:01,331 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:37:01,332 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2525 [2023-09-07 17:37:01,332 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:37:01,333 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:37:01,333 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:37:01,333 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:37:01,334 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:37:01,339 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:37:01,341 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:37:01.341806 [2023-09-07 17:37:01,478 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:37:01,508 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:37:01,540 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:37:01,543 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:37:01,550 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:37:01,551 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694108221.5067794, 'te': 1694108221.512733, 'initial_status': True} ====================================================================================================  [2023-09-07 17:37:01,552 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:37:01.552821 [2023-09-07 17:37:01,557 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:37:01,558 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:37:02,819 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:37:02,820 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:37:02,820 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:37:02,820 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:37:02,821 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:37:02,821 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:37:02,843 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:37:03,030 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:37:03,172 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:37:03,278 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:37:03,279 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2529 [2023-09-07 17:37:03,279 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:37:03,280 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:37:03,280 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:37:03,280 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:37:03,281 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:37:03,287 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:37:03,289 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:37:03.289220 [2023-09-07 17:37:03,559 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:37:03,587 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:37:03,616 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:37:03,622 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:37:03,629 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:37:03,629 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694108223.5845754, 'te': 1694108223.5921152, 'initial_status': True} ====================================================================================================  [2023-09-07 17:37:03,634 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:37:03.632029 [2023-09-07 17:37:03,641 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:37:03,642 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:37:10,034 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108220, schain_finish_ts: 1693742004 [2023-09-07 17:37:16,320 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108220, schain_finish_ts: 1692637668 [2023-09-07 17:37:17,767 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108232, schain_finish_ts: 0 [2023-09-07 17:37:19,305 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108232, schain_finish_ts: 1694108304 [2023-09-07 17:37:23,857 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108232, schain_finish_ts: 1692377364 [2023-09-07 17:37:29,873 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:37:29,873 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:37:29,875 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:37:29.875788 [2023-09-07 17:37:29,883 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:37:29,883 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:37:29,889 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:37:29,894 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694108249.8821816, 'te': 1694108249.8941221, 'initial_status': True} ====================================================================================================  [2023-09-07 17:37:29,896 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:37:29.896167 [2023-09-07 17:37:29,902 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:37:29,903 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:37:36,149 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:37:38,829 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:37:38,829 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:37:38,830 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:37:42,209 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:37:42,209 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:37:42,210 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:37:42,210 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:37:42,210 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:37:42,210 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:37:42,211 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:37:42,212 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:37:42,240 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:37:42,483 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:37:42,809 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:37:42,945 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:37:42,946 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2524 [2023-09-07 17:37:42,947 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:37:42,947 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:37:42,947 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:37:42,948 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:37:42,948 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:37:42,954 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:37:42,956 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:37:42.956560 [2023-09-07 17:37:43,138 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:37:43,166 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:37:43,194 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:37:43,196 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:37:43,209 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:37:43,210 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694108263.164767, 'te': 1694108263.170828, 'initial_status': True} ====================================================================================================  [2023-09-07 17:37:43,212 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:37:43.212194 [2023-09-07 17:37:43,246 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:37:43,248 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:37:49,703 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:37:49,703 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:37:49,704 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:37:51,347 INFO][1][MainProcess][MainThread] - core.schains.process_manager:109 - Get leaving_history for node ... [2023-09-07 17:37:53,432 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:37:53,433 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:37:53,433 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:37:53,434 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:37:53,435 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:37:53,435 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 17:37:53,467 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:37:53,642 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:37:53,642 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:37:53,643 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:37:53,644 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:37:53,644 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:37:53,645 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:37:53,645 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:37:53,659 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 17:37:53,715 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:37:53,902 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:37:54,027 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:37:54,279 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:37:54,294 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:37:54,294 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:37:54,297 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:37:54.297623 [2023-09-07 17:37:54,301 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108268, schain_finish_ts: 1693412520 [2023-09-07 17:37:54,310 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:37:54,310 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:37:54,317 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:37:54,323 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694108274.3079126, 'te': 1694108274.3235993, 'initial_status': True} ====================================================================================================  [2023-09-07 17:37:54,326 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:37:54.326608 [2023-09-07 17:37:54,332 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:37:54,332 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:37:54,389 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:37:54,390 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2521 [2023-09-07 17:37:54,391 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:37:54,391 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:37:54,392 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:37:54,392 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:37:54,393 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:37:54,399 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:37:54,401 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:37:54.401281 [2023-09-07 17:37:54,446 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:37:54,545 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:37:54,547 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2509 [2023-09-07 17:37:54,547 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:37:54,547 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:37:54,548 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:37:54,548 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:37:54,548 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:37:54,555 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:37:54,557 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:37:54.557240 [2023-09-07 17:37:54,578 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:37:54,604 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:37:54,631 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:37:54,633 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:37:54,640 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:37:54,640 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694108274.6027536, 'te': 1694108274.608756, 'initial_status': True} ====================================================================================================  [2023-09-07 17:37:54,642 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:37:54.642727 [2023-09-07 17:37:54,649 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:37:54,649 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:37:54,737 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:37:54,764 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:37:54,794 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:37:54,797 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:37:54,804 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:37:54,805 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694108274.7629206, 'te': 1694108274.7699027, 'initial_status': True} ====================================================================================================  [2023-09-07 17:37:54,807 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:37:54.807181 [2023-09-07 17:37:54,811 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:37:54,812 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:37:57,540 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108268, schain_finish_ts: 1693742304 [2023-09-07 17:38:25,959 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108292, schain_finish_ts: 1694108304 [2023-09-07 17:38:26,671 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:38:26,671 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:38:26,671 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:38:26,671 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:38:26,672 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:38:26,672 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:38:26,675 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:38:26,677 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:38:26,696 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:38:26,737 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:38:27,062 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:38:27,198 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:38:27,321 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:38:27,322 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2526 [2023-09-07 17:38:27,322 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:38:27,323 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:38:27,323 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:38:27,323 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:38:27,324 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:38:27,330 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:38:27,332 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:38:27.332665 [2023-09-07 17:38:27,511 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:38:27,541 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:38:27,578 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:38:27,581 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:38:27,592 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:38:27,592 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694108307.5394497, 'te': 1694108307.546495, 'initial_status': True} ====================================================================================================  [2023-09-07 17:38:27,594 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:38:27.594899 [2023-09-07 17:38:27,599 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:38:27,600 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:38:28,980 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108304, schain_finish_ts: 1692637668 [2023-09-07 17:38:33,482 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:38:33,483 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:38:33,483 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:38:33,483 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:38:33,483 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:38:33,484 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 17:38:33,509 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:38:33,696 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108304, schain_finish_ts: 1692377616 [2023-09-07 17:38:33,723 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:38:33,743 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:38:33,744 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:38:33,750 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 17:38:33,881 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:38:33,908 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:38:33,908 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:38:33,910 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:38:33.910552 [2023-09-07 17:38:33,917 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:38:33,917 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:38:33,923 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:38:33,928 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694108313.9163868, 'te': 1694108313.9286335, 'initial_status': True} ====================================================================================================  [2023-09-07 17:38:33,930 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:38:33.930646 [2023-09-07 17:38:33,935 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:38:33,936 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:38:34,006 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:38:34,007 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2530 [2023-09-07 17:38:34,008 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:38:34,008 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:38:34,008 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:38:34,008 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:38:34,009 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:38:34,014 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:38:34,017 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:38:34.017156 [2023-09-07 17:38:34,153 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:38:34,178 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:38:34,208 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:38:34,210 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:38:34,216 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:38:34,217 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694108314.1769996, 'te': 1694108314.1841476, 'initial_status': True} ====================================================================================================  [2023-09-07 17:38:34,218 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:38:34.218748 [2023-09-07 17:38:34,222 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:38:34,223 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:38:39,444 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108304, schain_finish_ts: 1692377364 [2023-09-07 17:38:39,511 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108304, schain_finish_ts: 0 [2023-09-07 17:38:41,002 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108304, schain_finish_ts: 1693742004 [2023-09-07 17:38:43,478 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:38:43,479 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:38:43,479 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:38:43,479 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:38:43,479 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:38:43,480 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:38:43,481 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:38:43,482 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:38:43,504 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:38:43,690 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:38:43,823 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:38:43,938 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:38:43,939 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2525 [2023-09-07 17:38:43,940 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:38:43,940 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:38:43,941 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:38:43,941 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:38:43,941 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:38:43,947 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:38:43,949 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:38:43.949682 [2023-09-07 17:38:44,099 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:38:44,131 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:38:44,177 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:38:44,179 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:38:44,186 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:38:44,187 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694108324.130307, 'te': 1694108324.1469424, 'initial_status': True} ====================================================================================================  [2023-09-07 17:38:44,188 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:38:44.188736 [2023-09-07 17:38:44,194 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:38:44,195 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:38:47,637 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:38:47,637 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:38:47,639 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:38:47.639331 [2023-09-07 17:38:47,646 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:38:47,647 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:38:47,653 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:38:47,656 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694108327.6451974, 'te': 1694108327.656694, 'initial_status': True} ====================================================================================================  [2023-09-07 17:38:47,658 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:38:47.658567 [2023-09-07 17:38:47,664 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:38:47,664 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:38:53,614 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108304, schain_finish_ts: 0 [2023-09-07 17:38:57,314 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 17:38:57,314 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 17:38:58,326 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:38:58,327 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:38:58,328 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:38:58,328 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:38:58,328 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:38:58,328 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:38:58,329 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:38:58,330 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 17:38:58,357 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:38:58,489 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:38:58,490 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:38:58,490 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:38:58,491 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:38:58,491 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:38:58,492 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 17:38:58,527 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:38:58,755 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:38:58,894 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:38:59,034 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:38:59,169 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:38:59,170 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2510 [2023-09-07 17:38:59,170 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:38:59,170 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:38:59,171 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:38:59,171 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:38:59,171 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:38:59,177 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:38:59,177 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:38:59,180 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:38:59.180022 [2023-09-07 17:38:59,328 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:38:59,331 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:38:59,333 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2522 [2023-09-07 17:38:59,334 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:38:59,334 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:38:59,335 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:38:59,335 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:38:59,336 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:38:59,344 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:38:59,347 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:38:59.346976 [2023-09-07 17:38:59,361 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:38:59,394 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:38:59,396 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:38:59,403 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:38:59,404 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694108339.35916, 'te': 1694108339.3655586, 'initial_status': True} ====================================================================================================  [2023-09-07 17:38:59,406 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:38:59.406500 [2023-09-07 17:38:59,411 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:38:59,411 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:38:59,534 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:38:59,559 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:38:59,591 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:38:59,593 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:38:59,601 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:38:59,602 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694108339.5570369, 'te': 1694108339.5653608, 'initial_status': True} ====================================================================================================  [2023-09-07 17:38:59,603 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:38:59.603871 [2023-09-07 17:38:59,611 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:38:59,611 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:38:59,696 INFO][1][MainProcess][MainThread] - core.schains.process_manager:117 - Got leaving sChains for the node: [] [2023-09-07 17:38:59,696 INFO][1][MainProcess][MainThread] - core.schains.process_manager:101 -  ==================================================================================================== Monitoring sChains Node ID: 16 sChains on node: [{'name': 'portly-passionate-sirius', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666281384, 'startBlock': 7803380, 'deposit': 0, 'index': 1, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x0000017723ca2', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'curly-red-alterf', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1676058936, 'startBlock': 8469646, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000001fe6cdfa', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'howling-hoedus', 'mainnetOwner': '0x1026467C91fdA40aCc7e6935d1E52f67274FD669', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666181400, 'startBlock': 7796660, 'deposit': 0, 'index': 0, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0xe5a600f4c2608', 'multitransactionMode': False, 'thresholdEncryption': False, 'active': True}, {'name': 'green-giddy-denebola', 'mainnetOwner': '0x8e835d5d9E28ffD259E8Fa8988Cacfc333A582B4', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674565164, 'startBlock': 8368049, 'deposit': 0, 'index': 4, 'generation': 1, 'originator': '0xd2001DAb6898127Be2F167B548691C87251D13C3', 'chainId': '0x00000585eb4b1', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'wan-tasty-kitalpha', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674568392, 'startBlock': 8368271, 'deposit': 0, 'index': 5, 'generation': 1, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x00000152e7be9', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'free-basic-rosy-SKALE', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1693996920, 'startBlock': 9646443, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000002bb90439', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'submissive-well-off-rotanev', 'mainnetOwner': '0x487360568fE5Ee4AE5c75782D5F63e9935B0E512', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1692361104, 'startBlock': 9539053, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000003791fa10', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}] Number of sChains on node: 7 Empty sChain structs: 0 ====================================================================================================  [2023-09-07 17:39:07,066 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:39:07,851 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:39:07,851 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:39:07,852 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:39:12,318 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:39:12,318 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain portly-passionate-sirius - Process is running: PID = 7 [2023-09-07 17:39:21,221 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:39:21,221 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain curly-red-alterf - Process is running: PID = 8 [2023-09-07 17:39:22,734 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108352, schain_finish_ts: 1693742304 [2023-09-07 17:39:24,704 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:39:24,705 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain howling-hoedus - Process is running: PID = 9 [2023-09-07 17:39:27,678 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108364, schain_finish_ts: 1692637668 [2023-09-07 17:39:29,247 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108364, schain_finish_ts: 1692377364 [2023-09-07 17:39:29,660 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:39:29,661 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain green-giddy-denebola - Process is running: PID = 10 [2023-09-07 17:39:41,769 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:39:41,770 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:39:41,772 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:39:41.772705 [2023-09-07 17:39:41,779 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:39:41,779 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:39:41,785 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:39:41,789 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694108381.7782285, 'te': 1694108381.7893045, 'initial_status': True} ====================================================================================================  [2023-09-07 17:39:41,791 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:39:41.791437 [2023-09-07 17:39:41,795 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:39:41,795 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:39:45,106 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:39:45,107 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain wan-tasty-kitalpha - Process is running: PID = 11 [2023-09-07 17:39:49,193 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108376, schain_finish_ts: 1693412520 [2023-09-07 17:39:49,899 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108364, schain_finish_ts: 1694108304 [2023-09-07 17:39:49,902 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108376, schain_finish_ts: 1693742004 [2023-09-07 17:39:51,481 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:39:51,481 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain free-basic-rosy-SKALE - Process is running: PID = 30622 [2023-09-07 17:39:52,657 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:39:52,658 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:39:52,660 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:39:52.660053 [2023-09-07 17:39:52,667 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:39:52,667 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:39:52,672 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:39:52,676 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694108392.665556, 'te': 1694108392.676107, 'initial_status': True} ====================================================================================================  [2023-09-07 17:39:52,678 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:39:52.678136 [2023-09-07 17:39:52,681 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:39:52,682 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:39:52,717 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108388, schain_finish_ts: 0 [2023-09-07 17:39:54,285 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:39:54,285 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:39:54,286 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:39:54,286 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:39:54,286 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:39:54,286 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 17:39:54,316 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:39:54,328 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:39:54,328 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:39:54,328 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:39:54,329 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:39:54,329 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:39:54,329 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 17:39:54,361 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:39:54,534 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:39:54,861 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:39:55,024 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:39:55,191 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:39:55,192 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:39:55,193 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2531 [2023-09-07 17:39:55,194 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:39:55,194 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:39:55,194 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:39:55,194 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:39:55,195 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:39:55,202 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:39:55,204 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:39:55.204763 [2023-09-07 17:39:55,295 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:39:55,296 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2526 [2023-09-07 17:39:55,297 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:39:55,297 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:39:55,297 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:39:55,298 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:39:55,298 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:39:55,304 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:39:55,306 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:39:55.306597 [2023-09-07 17:39:55,493 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:39:55,525 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:39:55,534 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:39:55,535 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain submissive-well-off-rotanev - Process is running: PID = 24912 [2023-09-07 17:39:55,535 INFO][1][MainProcess][MainThread] - core.schains.process_manager:88 - Process manager procedure finished [2023-09-07 17:39:55,535 INFO][1][MainProcess][MainThread] - __main__:65 - Sleeping for 90s after run_process_manager [2023-09-07 17:39:55,559 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:39:55,562 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:39:55,572 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:39:55,572 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694108395.5231295, 'te': 1694108395.5297582, 'initial_status': True} ====================================================================================================  [2023-09-07 17:39:55,574 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:39:55.574912 [2023-09-07 17:39:55,581 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:39:55,581 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:39:55,647 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:39:55,677 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:39:55,710 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:39:55,712 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:39:55,720 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:39:55,720 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694108395.6747723, 'te': 1694108395.6833854, 'initial_status': True} ====================================================================================================  [2023-09-07 17:39:55,722 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:39:55.722520 [2023-09-07 17:39:55,726 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:39:55,727 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:39:56,161 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:39:56,161 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:39:56,161 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:39:56,162 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:39:56,162 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:39:56,162 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:39:56,165 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:39:56,169 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:39:56,195 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:39:56,311 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:39:56,311 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:39:56,311 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:39:56,312 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:39:56,312 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:39:56,312 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:39:56,321 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:39:56,327 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 17:39:56,364 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:39:56,448 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:39:56,453 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 17:39:56,632 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:39:56,632 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:39:56,632 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:39:56,633 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:39:56,681 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:39:56,810 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:39:56,811 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:39:56,813 INFO][8][curly-red-alterf][T_0] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 17:39:56,817 ERROR][8][curly-red-alterf][T_0] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:39:56,817 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:39:56,902 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:39:57,023 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:39:57,024 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2511 [2023-09-07 17:39:57,025 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:39:57,025 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:39:57,025 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:39:57,025 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:39:57,026 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:39:57,032 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:39:57,034 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:39:57.034584 [2023-09-07 17:39:57,114 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:39:57,219 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:39:57,220 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2527 [2023-09-07 17:39:57,221 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:39:57,221 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:39:57,221 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:39:57,222 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:39:57,222 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:39:57,228 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:39:57,230 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:39:57.230529 [2023-09-07 17:39:57,265 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:39:57,295 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:39:57,341 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:39:57,343 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:39:57,354 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:39:57,354 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694108397.2933743, 'te': 1694108397.3056993, 'initial_status': True} ====================================================================================================  [2023-09-07 17:39:57,357 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:39:57.357017 [2023-09-07 17:39:57,361 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:39:57,362 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:39:57,422 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:39:57,461 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:39:57,496 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:39:57,498 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:39:57,504 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:39:57,505 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694108397.4564137, 'te': 1694108397.47081, 'initial_status': True} ====================================================================================================  [2023-09-07 17:39:57,507 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:39:57.507203 [2023-09-07 17:39:57,513 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:39:57,513 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:40:03,216 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:40:03,216 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:40:03,217 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:40:03,217 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 17:40:03,217 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 17:40:03,218 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:40:03,219 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:40:03,220 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 17:40:03,247 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:40:03,503 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:40:03,920 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:40:04,025 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:40:04,026 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2523 [2023-09-07 17:40:04,027 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:40:04,027 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:40:04,028 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:40:04,028 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:40:04,028 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:40:04,036 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:40:04,038 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:40:04.038064 [2023-09-07 17:40:04,195 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:40:04,220 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:40:04,252 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:40:04,255 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:40:04,263 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:40:04,264 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694108404.2188203, 'te': 1694108404.2261655, 'initial_status': True} ====================================================================================================  [2023-09-07 17:40:04,265 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:40:04.265700 [2023-09-07 17:40:04,270 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:40:04,271 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:40:04,651 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:40:04,701 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:40:04,701 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:40:04,701 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:40:11,383 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:40:11,387 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:40:11,392 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 17:40:22,535 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:40:22,535 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:40:22,539 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:40:22.539593 [2023-09-07 17:40:22,547 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:40:22,547 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:40:22,554 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:40:22,559 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694108422.545984, 'te': 1694108422.5588455, 'initial_status': True} ====================================================================================================  [2023-09-07 17:40:22,561 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:40:22.561479 [2023-09-07 17:40:22,566 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:40:22,567 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:40:30,113 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:40:30,113 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:40:30,117 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:40:30.117243 [2023-09-07 17:40:30,122 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:40:30,330 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:40:30,331 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:40:30,336 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:40:30.336046 [2023-09-07 17:40:30,348 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:40:30,348 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:40:30,354 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:40:30,359 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694108430.344046, 'te': 1694108430.3593621, 'initial_status': True} ====================================================================================================  [2023-09-07 17:40:30,361 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:40:30.361218 [2023-09-07 17:40:30,365 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:40:30,365 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:40:32,012 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108424, schain_finish_ts: 1692377364 [2023-09-07 17:40:35,703 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108424, schain_finish_ts: 1693412520 [2023-09-07 17:40:40,872 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108424, schain_finish_ts: 1692637668 [2023-09-07 17:40:42,152 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:40:42,364 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108436, schain_finish_ts: 1693742004 [2023-09-07 17:40:43,001 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108436, schain_finish_ts: 1693742304 [2023-09-07 17:40:45,614 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:40:45,614 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:40:45,614 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:40:45,615 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:40:45,615 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:40:45,615 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:40:45,617 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:40:45,618 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108436, schain_finish_ts: 1694108304 [2023-09-07 17:40:45,618 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:40:45,646 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:40:45,663 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:40:45,663 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:40:45,663 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:40:45,664 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:40:45,664 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:40:45,664 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:40:45,666 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:40:45,668 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:40:45,692 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:40:46,140 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:40:46,308 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:40:46,447 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:40:46,572 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:40:46,573 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2512 [2023-09-07 17:40:46,574 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:40:46,574 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:40:46,575 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:40:46,575 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:40:46,575 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:40:46,580 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:40:46,581 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:40:46,584 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:40:46.584108 [2023-09-07 17:40:46,687 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:40:46,688 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2528 [2023-09-07 17:40:46,689 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:40:46,689 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:40:46,689 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:40:46,689 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:40:46,690 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:40:46,695 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:40:46,697 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:40:46.697320 [2023-09-07 17:40:46,735 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:40:46,759 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:40:46,787 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:40:46,789 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:40:46,797 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:40:46,797 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694108446.7579703, 'te': 1694108446.7644017, 'initial_status': True} ====================================================================================================  [2023-09-07 17:40:46,799 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:40:46.799359 [2023-09-07 17:40:46,803 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:40:46,804 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:40:46,855 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:40:46,882 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:40:46,912 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:40:46,913 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:40:46,920 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:40:46,921 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694108446.8807929, 'te': 1694108446.887442, 'initial_status': True} ====================================================================================================  [2023-09-07 17:40:46,922 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:40:46.922600 [2023-09-07 17:40:46,926 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:40:46,927 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:40:47,722 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108436, schain_finish_ts: 0 [2023-09-07 17:40:48,791 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:40:48,791 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:40:48,791 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:40:48,791 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:40:48,792 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:40:48,792 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 17:40:48,817 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:40:49,131 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:40:49,356 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:40:49,458 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:40:49,459 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2524 [2023-09-07 17:40:49,460 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:40:49,460 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:40:49,461 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:40:49,461 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:40:49,461 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:40:49,468 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:40:49,470 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:40:49.470740 [2023-09-07 17:40:49,633 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:40:49,670 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:40:49,702 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:40:49,704 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:40:49,712 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:40:49,712 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694108449.6684127, 'te': 1694108449.677208, 'initial_status': True} ====================================================================================================  [2023-09-07 17:40:49,714 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:40:49.714783 [2023-09-07 17:40:49,719 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:40:49,720 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:40:52,703 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:40:52,704 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:40:52,705 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:40:52,706 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:40:52,706 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:40:52,706 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:40:52,713 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:40:52,714 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 17:40:52,749 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:40:52,984 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:40:53,128 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:40:53,248 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:40:53,249 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2527 [2023-09-07 17:40:53,250 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:40:53,250 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:40:53,251 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:40:53,251 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:40:53,251 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:40:53,258 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:40:53,260 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:40:53.260757 [2023-09-07 17:40:53,428 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:40:53,455 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:40:53,486 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:40:53,488 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:40:53,495 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:40:53,495 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694108453.4530816, 'te': 1694108453.4598174, 'initial_status': True} ====================================================================================================  [2023-09-07 17:40:53,497 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:40:53.497524 [2023-09-07 17:40:53,501 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:40:53,502 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:40:54,389 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:40:54,389 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:40:54,390 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:40:54,390 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:40:54,390 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:40:54,391 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 17:40:54,414 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:40:54,648 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:40:54,811 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:40:54,905 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:40:54,906 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2532 [2023-09-07 17:40:54,907 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:40:54,907 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:40:54,908 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:40:54,908 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:40:54,909 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:40:54,915 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:40:54,917 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:40:54.917831 [2023-09-07 17:40:55,193 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:40:55,222 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:40:55,250 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:40:55,252 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:40:55,261 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:40:55,262 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694108455.220688, 'te': 1694108455.227865, 'initial_status': True} ====================================================================================================  [2023-09-07 17:40:55,263 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:40:55.263746 [2023-09-07 17:40:55,268 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:40:55,268 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:40:57,954 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:40:57,954 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:40:57,958 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:40:57.958663 [2023-09-07 17:40:57,968 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:40:57,968 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:40:57,978 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:40:57,984 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694108457.9663124, 'te': 1694108457.9845593, 'initial_status': True} ====================================================================================================  [2023-09-07 17:40:57,987 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:40:57.987238 [2023-09-07 17:40:57,993 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:40:57,993 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:40:58,460 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:40:58,460 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:40:58,461 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:40:59,644 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:40:59,645 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:40:59,645 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 17:40:59,807 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:40:59,808 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:40:59,809 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:40:59.809950 [2023-09-07 17:40:59,817 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:40:59,818 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:40:59,826 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:40:59,832 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694108459.815997, 'te': 1694108459.8323371, 'initial_status': True} ====================================================================================================  [2023-09-07 17:40:59,836 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:40:59.836508 [2023-09-07 17:40:59,842 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:40:59,843 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:41:21,056 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:41:23,066 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:41:23,066 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:41:23,069 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:41:23.069095 [2023-09-07 17:41:23,079 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:41:23,079 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:41:23,087 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:41:23,093 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694108483.0779526, 'te': 1694108483.0934422, 'initial_status': True} ====================================================================================================  [2023-09-07 17:41:23,095 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:41:23.095769 [2023-09-07 17:41:23,101 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:41:23,101 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:41:23,900 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108460, schain_finish_ts: 1693412520 [2023-09-07 17:41:25,614 INFO][1][MainProcess][MainThread] - core.schains.cleaner:65 - Cleaner process started [2023-09-07 17:41:25,645 INFO][30809][cleaner][MainThread] - core.schains.cleaner:110 - Cleaner procedure started. [2023-09-07 17:41:34,297 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:41:34,297 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:41:34,297 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:41:34,297 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:41:34,298 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:41:34,298 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:41:34,299 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:41:34,300 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:41:34,326 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:41:34,492 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:41:34,619 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:41:34,719 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:41:34,720 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2529 [2023-09-07 17:41:34,721 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:41:34,721 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:41:34,721 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:41:34,722 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:41:34,722 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:41:34,732 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:41:34,734 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:41:34.734961 [2023-09-07 17:41:35,034 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:41:35,064 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:41:35,096 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:41:35,098 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:41:35,105 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:41:35,105 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694108495.0628104, 'te': 1694108495.0704837, 'initial_status': True} ====================================================================================================  [2023-09-07 17:41:35,107 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:41:35.107765 [2023-09-07 17:41:35,111 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:41:35,112 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:41:37,275 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108484, schain_finish_ts: 1692377364 [2023-09-07 17:41:41,879 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108496, schain_finish_ts: 1693742304 [2023-09-07 17:41:43,114 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108496, schain_finish_ts: 1693742004 [2023-09-07 17:41:43,499 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108496, schain_finish_ts: 1694108304 [2023-09-07 17:41:43,823 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108496, schain_finish_ts: 1692637668 [2023-09-07 17:41:49,799 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:41:49,800 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:41:49,800 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:41:50,729 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:41:50,729 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:41:50,729 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:41:50,729 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:41:50,730 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:41:50,730 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:41:50,753 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:41:50,953 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:41:51,072 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108496, schain_finish_ts: 0 [2023-09-07 17:41:51,112 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:41:51,229 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:41:51,231 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2533 [2023-09-07 17:41:51,231 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:41:51,231 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:41:51,232 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:41:51,232 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:41:51,232 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:41:51,238 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:41:51,240 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:41:51.240477 [2023-09-07 17:41:51,472 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:41:51,499 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:41:51,536 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:41:51,538 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:41:51,544 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:41:51,545 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694108511.4976645, 'te': 1694108511.504643, 'initial_status': True} ====================================================================================================  [2023-09-07 17:41:51,546 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:41:51.546800 [2023-09-07 17:41:51,550 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:41:51,551 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:41:55,959 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:41:55,959 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:41:55,960 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:41:55,960 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:41:55,960 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:41:55,960 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 17:41:55,991 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:41:56,286 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:41:56,431 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:41:56,593 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:41:56,595 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2525 [2023-09-07 17:41:56,596 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:41:56,596 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:41:56,596 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:41:56,597 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:41:56,597 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:41:56,606 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:41:56,610 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:41:56.610219 [2023-09-07 17:41:56,795 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:41:56,832 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:41:56,870 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:41:56,872 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:41:56,881 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:41:56,881 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694108516.8302536, 'te': 1694108516.8392875, 'initial_status': True} ====================================================================================================  [2023-09-07 17:41:56,883 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:41:56.883784 [2023-09-07 17:41:56,888 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:41:56,889 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:42:00,070 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:42:00,070 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:42:00,070 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:42:00,070 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:42:00,071 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:42:00,071 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:42:00,073 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:42:00,073 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:42:00,117 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:42:00,308 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:42:00,308 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:42:00,309 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:42:00,309 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:42:00,309 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:42:00,310 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:42:00,317 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:42:00,321 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:42:00,336 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:42:00,354 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:42:00,377 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:42:00,513 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:42:00,621 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:42:00,622 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2513 [2023-09-07 17:42:00,622 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:42:00,623 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:42:00,623 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:42:00,623 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:42:00,624 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:42:00,630 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:42:00,632 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:42:00.632689 [2023-09-07 17:42:00,801 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:42:00,933 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:42:00,968 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:42:01,010 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:42:01,012 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:42:01,020 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:42:01,020 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694108520.9641953, 'te': 1694108520.9751248, 'initial_status': True} ====================================================================================================  [2023-09-07 17:42:01,022 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:42:01.022427 [2023-09-07 17:42:01,027 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:42:01,027 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:42:01,107 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:42:01,211 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:42:01,212 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2528 [2023-09-07 17:42:01,213 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:42:01,213 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:42:01,213 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:42:01,213 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:42:01,214 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:42:01,219 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:42:01,221 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:42:01.221355 [2023-09-07 17:42:01,357 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:42:01,381 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:42:01,416 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:42:01,419 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:42:01,426 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:42:01,427 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694108521.380155, 'te': 1694108521.3880188, 'initial_status': True} ====================================================================================================  [2023-09-07 17:42:01,428 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:42:01.428725 [2023-09-07 17:42:01,433 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:42:01,433 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:42:03,425 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:42:03,427 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:42:03,428 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:42:13,794 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:42:13,794 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:42:13,796 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:42:13.796133 [2023-09-07 17:42:13,802 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:42:13,802 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:42:13,808 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:42:13,812 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694108533.8012083, 'te': 1694108533.8122146, 'initial_status': True} ====================================================================================================  [2023-09-07 17:42:13,814 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:42:13.814189 [2023-09-07 17:42:13,818 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:42:13,818 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:42:14,647 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:42:14,647 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:42:14,649 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:42:14.649088 [2023-09-07 17:42:14,656 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:42:14,656 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:42:14,661 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:42:14,665 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694108534.6550586, 'te': 1694108534.6656528, 'initial_status': True} ====================================================================================================  [2023-09-07 17:42:14,667 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:42:14.667365 [2023-09-07 17:42:14,671 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:42:14,671 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:42:21,438 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:42:21,439 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:42:21,444 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:42:21.444785 [2023-09-07 17:42:21,454 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:42:21,454 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:42:21,467 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:42:21,472 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694108541.451807, 'te': 1694108541.472539, 'initial_status': True} ====================================================================================================  [2023-09-07 17:42:21,475 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:42:21.475348 [2023-09-07 17:42:21,480 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:42:21,480 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:42:22,936 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108532, schain_finish_ts: 1693412520 [2023-09-07 17:42:23,061 INFO][30809][cleaner][MainThread] - core.schains.cleaner:116 - sChains on contracts: ['portly-passionate-sirius', 'curly-red-alterf', 'howling-hoedus', 'green-giddy-denebola', 'wan-tasty-kitalpha', 'free-basic-rosy-SKALE', 'submissive-well-off-rotanev'] sChains on node: ['curly-red-alterf', 'free-basic-rosy-SKALE', 'green-giddy-denebola', 'howling-hoedus', 'portly-passionate-sirius', 'submissive-well-off-rotanev', 'wan-tasty-kitalpha'] [2023-09-07 17:42:23,061 INFO][30809][cleaner][MainThread] - core.schains.cleaner:132 - Cleanup procedure finished [2023-09-07 17:42:23,066 INFO][1][MainProcess][MainThread] - core.schains.cleaner:67 - Cleaner process is joined. [2023-09-07 17:42:23,066 INFO][1][MainProcess][MainThread] - core.schains.cleaner:68 - Terminating the process [2023-09-07 17:42:23,067 INFO][1][MainProcess][MainThread] - __main__:70 - Sleeping for 90s after run_cleaner [2023-09-07 17:42:26,545 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108532, schain_finish_ts: 1693742304 [2023-09-07 17:42:32,768 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108544, schain_finish_ts: 1692637668 [2023-09-07 17:42:34,216 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108544, schain_finish_ts: 1693742004 [2023-09-07 17:42:40,928 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108556, schain_finish_ts: 0 [2023-09-07 17:42:47,653 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108556, schain_finish_ts: 1694108304 [2023-09-07 17:42:48,349 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:42:52,709 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108568, schain_finish_ts: 1692377364 [2023-09-07 17:42:52,710 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:42:52,711 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:42:52,711 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:42:52,711 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:42:52,712 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:42:52,712 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 17:42:52,757 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:42:52,933 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:42:53,140 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:42:53,252 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:42:53,253 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2534 [2023-09-07 17:42:53,254 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:42:53,254 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:42:53,255 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:42:53,255 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:42:53,256 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:42:53,265 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:42:53,268 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:42:53.268414 [2023-09-07 17:42:53,360 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:42:53,360 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:42:53,360 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:42:53,361 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:42:53,361 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:42:53,362 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:42:53,364 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:42:53,364 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 17:42:53,393 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:42:53,416 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:42:53,441 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:42:53,470 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:42:53,470 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:42:53,471 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:42:53,471 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:42:53,471 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:42:53,471 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:42:53,472 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:42:53,472 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:42:53,473 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:42:53,475 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:42:53,481 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:42:53,482 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694108573.4397304, 'te': 1694108573.4472456, 'initial_status': True} ====================================================================================================  [2023-09-07 17:42:53,484 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:42:53.484316 [2023-09-07 17:42:53,490 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:42:53,490 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:42:53,500 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:42:53,576 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:42:53,695 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:42:53,899 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:42:54,024 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:42:54,025 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2530 [2023-09-07 17:42:54,026 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:42:54,026 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:42:54,026 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:42:54,027 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:42:54,027 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:42:54,033 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:42:54,035 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:42:54.035553 [2023-09-07 17:42:54,057 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:42:54,173 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:42:54,175 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2529 [2023-09-07 17:42:54,176 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:42:54,176 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:42:54,177 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:42:54,177 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:42:54,178 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:42:54,187 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:42:54,190 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:42:54.190606 [2023-09-07 17:42:54,203 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:42:54,232 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:42:54,265 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:42:54,266 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:42:54,274 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:42:54,275 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694108574.2299256, 'te': 1694108574.23821, 'initial_status': True} ====================================================================================================  [2023-09-07 17:42:54,276 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:42:54.276713 [2023-09-07 17:42:54,281 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:42:54,283 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:42:54,429 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:42:54,458 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:42:54,486 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:42:54,488 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:42:54,496 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:42:54,496 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694108574.4563477, 'te': 1694108574.4635513, 'initial_status': True} ====================================================================================================  [2023-09-07 17:42:54,498 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:42:54.498570 [2023-09-07 17:42:54,503 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:42:54,503 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:43:05,374 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:43:05,374 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:43:05,375 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 17:43:05,897 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:43:05,897 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:43:05,898 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 17:43:07,229 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:43:07,229 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:43:07,229 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:43:07,230 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:43:07,230 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:43:07,230 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 17:43:07,253 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:43:07,426 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:43:07,567 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:43:07,668 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:43:07,669 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2526 [2023-09-07 17:43:07,670 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:43:07,670 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:43:07,671 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:43:07,671 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:43:07,671 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:43:07,678 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:43:07,680 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:43:07.680078 [2023-09-07 17:43:07,870 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:43:07,894 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:43:07,922 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:43:07,924 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:43:07,931 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:43:07,931 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694108587.8929665, 'te': 1694108587.898827, 'initial_status': True} ====================================================================================================  [2023-09-07 17:43:07,933 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:43:07.933422 [2023-09-07 17:43:07,937 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:43:07,938 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:43:10,325 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:43:10,325 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:43:10,327 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:43:10.327755 [2023-09-07 17:43:10,334 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:43:10,334 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:43:10,340 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:43:10,344 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694108590.3333652, 'te': 1694108590.344232, 'initial_status': True} ====================================================================================================  [2023-09-07 17:43:10,345 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:43:10.345926 [2023-09-07 17:43:10,349 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:43:10,350 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:43:11,736 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:43:11,736 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:43:11,736 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:43:11,736 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:43:11,737 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:43:11,737 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:43:11,738 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:43:11,741 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:43:11,761 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:43:12,054 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:43:12,197 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:43:12,299 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:43:12,300 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2514 [2023-09-07 17:43:12,301 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:43:12,301 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:43:12,301 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:43:12,302 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:43:12,302 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:43:12,308 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:43:12,311 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:43:12.311002 [2023-09-07 17:43:12,458 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:43:12,484 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:43:12,514 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:43:12,516 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:43:12,524 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:43:12,524 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694108592.4826899, 'te': 1694108592.493063, 'initial_status': True} ====================================================================================================  [2023-09-07 17:43:12,526 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:43:12.526369 [2023-09-07 17:43:12,531 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:43:12,532 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:43:30,074 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:43:30,074 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:43:30,076 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:43:30.076365 [2023-09-07 17:43:30,083 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:43:30,083 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:43:30,090 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:43:30,096 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694108610.0823863, 'te': 1694108610.0958545, 'initial_status': True} ====================================================================================================  [2023-09-07 17:43:30,099 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:43:30.099088 [2023-09-07 17:43:30,105 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:43:30,105 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:43:31,995 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:43:31,995 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:43:31,998 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:43:31.998159 [2023-09-07 17:43:32,007 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:43:32,007 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:43:32,016 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:43:32,021 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694108612.0057719, 'te': 1694108612.0211298, 'initial_status': True} ====================================================================================================  [2023-09-07 17:43:32,023 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:43:32.023891 [2023-09-07 17:43:32,029 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:43:32,029 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:43:32,199 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108604, schain_finish_ts: 1693412520 [2023-09-07 17:43:32,335 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108604, schain_finish_ts: 0 [2023-09-07 17:43:37,117 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108616, schain_finish_ts: 1693742004 [2023-09-07 17:43:38,803 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108604, schain_finish_ts: 1693742304 [2023-09-07 17:43:38,867 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:43:44,261 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:43:44,261 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:43:44,263 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:43:44,264 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:43:44,266 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:43:44,267 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:43:44,273 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:43:44,276 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 17:43:44,303 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:43:44,320 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108616, schain_finish_ts: 1692637668 [2023-09-07 17:43:44,400 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:43:44,400 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:43:44,400 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:43:44,400 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:43:44,401 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:43:44,401 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:43:44,426 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:43:44,546 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:43:44,799 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:43:45,040 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:43:45,170 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:43:45,171 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:43:45,172 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2531 [2023-09-07 17:43:45,173 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:43:45,173 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:43:45,174 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:43:45,174 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:43:45,175 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:43:45,181 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:43:45,183 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:43:45.183159 [2023-09-07 17:43:45,276 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:43:45,278 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2527 [2023-09-07 17:43:45,279 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:43:45,279 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:43:45,279 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:43:45,280 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:43:45,280 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:43:45,287 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:43:45,289 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:43:45.289390 [2023-09-07 17:43:45,375 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:43:45,406 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:43:45,440 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:43:45,442 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:43:45,449 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:43:45,450 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694108625.40465, 'te': 1694108625.410889, 'initial_status': True} ====================================================================================================  [2023-09-07 17:43:45,451 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:43:45.451818 [2023-09-07 17:43:45,456 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:43:45,456 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:43:45,539 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:43:45,566 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:43:45,603 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:43:45,605 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:43:45,612 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:43:45,612 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694108625.5637877, 'te': 1694108625.5709777, 'initial_status': True} ====================================================================================================  [2023-09-07 17:43:45,613 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:43:45.613843 [2023-09-07 17:43:45,618 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:43:45,619 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:43:46,108 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108616, schain_finish_ts: 1694108304 [2023-09-07 17:43:46,132 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:43:46,132 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:43:46,133 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 17:43:46,133 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108616, schain_finish_ts: 1692377364 [2023-09-07 17:43:51,686 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:43:51,686 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:43:51,686 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:43:51,686 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:43:51,686 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:43:51,687 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 17:43:51,707 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:43:51,882 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:43:52,009 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:43:52,206 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:43:52,207 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2535 [2023-09-07 17:43:52,208 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:43:52,208 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:43:52,208 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:43:52,208 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:43:52,209 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:43:52,215 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:43:52,217 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:43:52.217630 [2023-09-07 17:43:52,378 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:43:52,402 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:43:52,431 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:43:52,433 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:43:52,440 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:43:52,440 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694108632.4009838, 'te': 1694108632.4079196, 'initial_status': True} ====================================================================================================  [2023-09-07 17:43:52,442 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:43:52.442344 [2023-09-07 17:43:52,447 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:43:52,448 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:43:53,055 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 17:43:53,150 INFO][1][MainProcess][MainThread] - core.schains.process_manager:57 - Process manager started [2023-09-07 17:43:53,222 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:43:53,222 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:43:53,222 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:43:53,223 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:43:53,406 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:43:53,407 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:43:53,410 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 17:43:53,417 ERROR][11][wan-tasty-kitalpha][T_0] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:43:53,418 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:43:57,953 INFO][1][MainProcess][MainThread] - core.schains.notifications:35 - Node account has 406903463189434189 WEI [2023-09-07 17:43:57,956 INFO][1][MainProcess][MainThread] - tools.notifications.messages:188 - Saving new balance state 3014 1 [2023-09-07 17:43:57,957 INFO][1][MainProcess][MainThread] - core.schains.process_manager:95 - Fetching schains to monitor... [2023-09-07 17:43:58,177 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:43:58,177 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:43:58,177 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:43:58,178 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:43:58,178 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:43:58,178 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:43:58,182 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:43:58,183 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 17:43:58,211 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:43:58,433 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:43:58,585 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:43:58,717 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:43:58,720 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2530 [2023-09-07 17:43:58,721 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:43:58,722 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:43:58,722 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:43:58,722 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:43:58,723 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:43:58,729 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:43:58,730 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:43:58.730825 [2023-09-07 17:43:58,914 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:43:58,939 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:43:58,965 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:43:58,967 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:43:58,973 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:43:58,974 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694108638.938262, 'te': 1694108638.9437914, 'initial_status': True} ====================================================================================================  [2023-09-07 17:43:58,975 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:43:58.975778 [2023-09-07 17:43:58,979 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:43:58,980 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:44:00,685 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:44:00,685 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:44:00,687 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:44:00.687745 [2023-09-07 17:44:00,694 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:44:00,694 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:44:00,696 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:44:00,697 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:44:00,698 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 17:44:00,702 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:44:00,707 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694108640.692971, 'te': 1694108640.70741, 'initial_status': True} ====================================================================================================  [2023-09-07 17:44:00,710 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:44:00.710127 [2023-09-07 17:44:00,715 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:44:00,715 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:44:09,676 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:44:09,677 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:44:09,677 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:44:09,677 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:44:09,678 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:44:09,678 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:44:09,682 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:44:09,689 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:44:09,739 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:44:10,036 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:44:10,166 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:44:10,289 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:44:10,290 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2515 [2023-09-07 17:44:10,291 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:44:10,292 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:44:10,292 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:44:10,293 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:44:10,294 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:44:10,301 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:44:10,303 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:44:10.303898 [2023-09-07 17:44:10,455 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:44:10,489 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:44:10,520 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:44:10,522 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:44:10,530 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:44:10,530 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694108650.486371, 'te': 1694108650.4940023, 'initial_status': True} ====================================================================================================  [2023-09-07 17:44:10,532 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:44:10.532278 [2023-09-07 17:44:10,538 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:44:10,538 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:44:24,932 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:44:25,223 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:44:25,224 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:44:25,226 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:44:25.226199 [2023-09-07 17:44:25,235 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:44:25,235 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:44:25,241 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:44:25,245 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694108665.233663, 'te': 1694108665.2456434, 'initial_status': True} ====================================================================================================  [2023-09-07 17:44:25,247 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:44:25.247589 [2023-09-07 17:44:25,252 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:44:25,253 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:44:31,324 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108664, schain_finish_ts: 0 [2023-09-07 17:44:31,533 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:44:31,533 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:44:31,535 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:44:31.535565 [2023-09-07 17:44:31,542 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:44:31,542 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:44:31,551 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:44:31,558 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694108671.5408163, 'te': 1694108671.5579205, 'initial_status': True} ====================================================================================================  [2023-09-07 17:44:31,560 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:44:31.560408 [2023-09-07 17:44:31,565 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:44:31,565 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:44:31,766 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108664, schain_finish_ts: 1693742004 [2023-09-07 17:44:32,076 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108664, schain_finish_ts: 1693742304 [2023-09-07 17:44:32,126 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108664, schain_finish_ts: 1693412520 [2023-09-07 17:44:38,887 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108676, schain_finish_ts: 1692637668 [2023-09-07 17:44:41,174 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 17:44:41,174 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 17:44:44,436 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:44:44,436 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:44:44,437 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:44:44,437 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:44:44,437 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:44:44,437 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 17:44:44,460 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:44:44,699 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:44:44,737 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108676, schain_finish_ts: 1694108304 [2023-09-07 17:44:44,803 INFO][1][MainProcess][MainThread] - core.schains.process_manager:109 - Get leaving_history for node ... [2023-09-07 17:44:44,856 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:44:44,957 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:44:44,960 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2528 [2023-09-07 17:44:44,960 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:44:44,961 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:44:44,961 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:44:44,961 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:44:44,961 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:44:44,968 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:44:44,970 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:44:44.970148 [2023-09-07 17:44:45,125 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:44:45,148 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:44:45,169 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:44:45,170 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:44:45,170 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:44:45,170 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:44:45,170 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:44:45,171 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:44:45,173 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:44:45,174 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 17:44:45,176 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:44:45,178 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:44:45,188 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:44:45,188 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694108685.1470578, 'te': 1694108685.1533906, 'initial_status': True} ====================================================================================================  [2023-09-07 17:44:45,190 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:44:45.190530 [2023-09-07 17:44:45,194 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:44:45,195 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:44:45,199 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:44:45,379 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:44:45,513 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:44:45,524 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:44:45,524 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:44:45,524 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:44:45,525 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 17:44:45,525 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 17:44:45,525 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:44:45,527 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:44:45,531 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:44:45,570 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:44:45,662 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:44:45,663 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2531 [2023-09-07 17:44:45,664 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:44:45,664 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:44:45,664 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:44:45,664 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:44:45,665 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:44:45,670 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:44:45,672 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:44:45.672497 [2023-09-07 17:44:45,771 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:44:46,008 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:44:46,041 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:44:46,072 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:44:46,074 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:44:46,081 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:44:46,082 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694108686.040076, 'te': 1694108686.0460258, 'initial_status': True} ====================================================================================================  [2023-09-07 17:44:46,083 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:44:46.083541 [2023-09-07 17:44:46,087 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:44:46,088 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:44:46,171 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:44:46,274 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:44:46,275 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2536 [2023-09-07 17:44:46,276 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:44:46,276 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:44:46,276 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:44:46,277 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:44:46,277 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:44:46,283 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:44:46,287 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:44:46.287428 [2023-09-07 17:44:46,431 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:44:46,474 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:44:46,511 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:44:46,513 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:44:46,523 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:44:46,524 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694108686.472084, 'te': 1694108686.4806316, 'initial_status': True} ====================================================================================================  [2023-09-07 17:44:46,526 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:44:46.526760 [2023-09-07 17:44:46,532 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:44:46,533 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:44:47,519 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:44:47,519 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:44:47,520 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 17:44:47,525 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:44:47,525 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:44:47,526 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:44:47,527 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:44:47,527 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:44:47,527 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:44:47,534 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:44:47,535 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 17:44:47,570 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:44:47,789 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:44:47,938 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:44:48,071 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:44:48,072 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2532 [2023-09-07 17:44:48,073 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:44:48,073 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:44:48,073 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:44:48,074 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:44:48,074 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:44:48,080 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:44:48,082 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:44:48.082693 [2023-09-07 17:44:48,221 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:44:48,248 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:44:48,277 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:44:48,279 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:44:48,286 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:44:48,287 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694108688.247061, 'te': 1694108688.253568, 'initial_status': True} ====================================================================================================  [2023-09-07 17:44:48,288 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:44:48.288665 [2023-09-07 17:44:48,292 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:44:48,293 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:44:52,936 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:44:52,937 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:44:52,937 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:44:57,453 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108688, schain_finish_ts: 1692377364 [2023-09-07 17:45:00,470 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:45:00,927 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108688, schain_finish_ts: 1692377616 [2023-09-07 17:45:01,601 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:45:01,601 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:45:01,605 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:45:01.605668 [2023-09-07 17:45:01,617 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:45:06,538 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:45:06,538 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:45:06,541 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:45:06.541344 [2023-09-07 17:45:06,550 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:45:06,550 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:45:06,557 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:45:06,564 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694108706.5481772, 'te': 1694108706.5643194, 'initial_status': True} ====================================================================================================  [2023-09-07 17:45:06,573 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:45:06.573377 [2023-09-07 17:45:06,582 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:45:06,583 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:45:07,870 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:45:07,870 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:45:07,873 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:45:07.873742 [2023-09-07 17:45:07,885 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:45:07,886 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:45:07,896 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:45:07,906 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694108707.882717, 'te': 1694108707.9060602, 'initial_status': True} ====================================================================================================  [2023-09-07 17:45:07,908 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:45:07.908512 [2023-09-07 17:45:07,914 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:45:07,914 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:45:12,947 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:45:12,947 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:45:12,948 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:45:12,948 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:45:12,948 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:45:12,949 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:45:12,951 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:45:12,952 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:45:12,983 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:45:13,313 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:45:13,469 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:45:13,581 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:45:13,582 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2516 [2023-09-07 17:45:13,582 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:45:13,583 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:45:13,583 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:45:13,583 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:45:13,583 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:45:13,590 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:45:13,592 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:45:13.592105 [2023-09-07 17:45:13,737 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:45:13,768 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:45:13,806 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:45:13,809 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:45:13,819 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:45:13,820 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694108713.7666154, 'te': 1694108713.7751603, 'initial_status': True} ====================================================================================================  [2023-09-07 17:45:13,821 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:45:13.821880 [2023-09-07 17:45:13,826 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:45:13,827 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:45:19,114 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108712, schain_finish_ts: 1692637668 [2023-09-07 17:45:20,762 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108712, schain_finish_ts: 0 [2023-09-07 17:45:26,718 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108712, schain_finish_ts: 1693742004 [2023-09-07 17:45:29,908 INFO][1][MainProcess][MainThread] - core.schains.process_manager:117 - Got leaving sChains for the node: [] [2023-09-07 17:45:29,909 INFO][1][MainProcess][MainThread] - core.schains.process_manager:101 -  ==================================================================================================== Monitoring sChains Node ID: 16 sChains on node: [{'name': 'portly-passionate-sirius', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666281384, 'startBlock': 7803380, 'deposit': 0, 'index': 1, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x0000017723ca2', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'curly-red-alterf', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1676058936, 'startBlock': 8469646, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000001fe6cdfa', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'howling-hoedus', 'mainnetOwner': '0x1026467C91fdA40aCc7e6935d1E52f67274FD669', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666181400, 'startBlock': 7796660, 'deposit': 0, 'index': 0, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0xe5a600f4c2608', 'multitransactionMode': False, 'thresholdEncryption': False, 'active': True}, {'name': 'green-giddy-denebola', 'mainnetOwner': '0x8e835d5d9E28ffD259E8Fa8988Cacfc333A582B4', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674565164, 'startBlock': 8368049, 'deposit': 0, 'index': 4, 'generation': 1, 'originator': '0xd2001DAb6898127Be2F167B548691C87251D13C3', 'chainId': '0x00000585eb4b1', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'wan-tasty-kitalpha', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674568392, 'startBlock': 8368271, 'deposit': 0, 'index': 5, 'generation': 1, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x00000152e7be9', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'free-basic-rosy-SKALE', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1693996920, 'startBlock': 9646443, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000002bb90439', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'submissive-well-off-rotanev', 'mainnetOwner': '0x487360568fE5Ee4AE5c75782D5F63e9935B0E512', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1692361104, 'startBlock': 9539053, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000003791fa10', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}] Number of sChains on node: 7 Empty sChain structs: 0 ====================================================================================================  [2023-09-07 17:45:30,163 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108724, schain_finish_ts: 0 [2023-09-07 17:45:32,863 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108724, schain_finish_ts: 1694108304 [2023-09-07 17:45:33,506 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:45:33,506 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain portly-passionate-sirius - Process is running: PID = 7 [2023-09-07 17:45:33,613 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108724, schain_finish_ts: 1693742304 [2023-09-07 17:45:33,674 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108724, schain_finish_ts: 1693412520 [2023-09-07 17:45:35,482 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:45:35,482 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:45:35,483 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:45:35,483 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:45:35,483 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:45:35,483 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:45:35,485 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:45:35,485 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:45:35,516 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:45:35,733 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:45:35,890 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:45:35,994 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:45:35,995 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2532 [2023-09-07 17:45:35,996 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:45:35,996 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:45:35,996 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:45:35,996 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:45:35,997 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:45:36,002 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:45:36,004 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:45:36.004545 [2023-09-07 17:45:36,134 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:45:36,179 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:45:36,206 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:45:36,208 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:45:36,212 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:45:36,215 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:45:36,215 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694108736.1780403, 'te': 1694108736.184196, 'initial_status': True} ====================================================================================================  [2023-09-07 17:45:36,217 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:45:36.217291 [2023-09-07 17:45:36,221 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:45:36,222 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:45:36,305 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:45:36,305 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain curly-red-alterf - Process is running: PID = 8 [2023-09-07 17:45:36,307 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:45:36,308 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:45:36,308 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:45:36,308 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:45:36,308 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:45:36,309 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:45:36,331 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:45:36,662 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:45:36,821 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:45:36,920 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:45:36,922 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2529 [2023-09-07 17:45:36,923 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:45:36,923 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:45:36,924 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:45:36,924 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:45:36,925 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:45:36,932 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:45:36,934 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:45:36.934404 [2023-09-07 17:45:37,104 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:45:37,130 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:45:37,159 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:45:37,161 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:45:37,169 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:45:37,169 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694108737.128989, 'te': 1694108737.1354306, 'initial_status': True} ====================================================================================================  [2023-09-07 17:45:37,172 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:45:37.172245 [2023-09-07 17:45:37,176 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:45:37,177 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:45:38,304 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:45:38,304 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain howling-hoedus - Process is running: PID = 9 [2023-09-07 17:45:47,143 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:45:47,144 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:45:47,144 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:45:49,270 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:45:49,271 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:45:49,272 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:45:49.272936 [2023-09-07 17:45:49,282 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:45:49,282 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:45:49,290 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:45:49,297 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694108749.2791204, 'te': 1694108749.2953117, 'initial_status': True} ====================================================================================================  [2023-09-07 17:45:49,300 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:45:49.300656 [2023-09-07 17:45:49,306 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:45:49,306 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:45:49,352 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:45:49,352 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:45:49,352 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 17:45:50,724 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:45:50,724 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain green-giddy-denebola - Process is running: PID = 10 [2023-09-07 17:45:57,328 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:45:57,328 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:45:57,328 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:45:57,329 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:45:57,329 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:45:57,329 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:45:57,331 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:45:57,331 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:45:57,349 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:45:57,349 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain wan-tasty-kitalpha - Process is running: PID = 11 [2023-09-07 17:45:57,363 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:45:57,656 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:45:57,805 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:45:57,921 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:45:57,922 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2533 [2023-09-07 17:45:57,923 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:45:57,923 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:45:57,923 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:45:57,924 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:45:57,924 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:45:57,930 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:45:57,932 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:45:57.932776 [2023-09-07 17:45:58,076 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:45:58,102 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:45:58,132 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:45:58,134 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:45:58,141 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:45:58,141 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694108758.100103, 'te': 1694108758.106813, 'initial_status': True} ====================================================================================================  [2023-09-07 17:45:58,143 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:45:58.143367 [2023-09-07 17:45:58,148 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:45:58,148 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:46:05,604 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:46:05,605 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:46:05,606 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:46:05.606832 [2023-09-07 17:46:05,614 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:46:05,614 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:46:05,620 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:46:05,625 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694108765.6129646, 'te': 1694108765.625236, 'initial_status': True} ====================================================================================================  [2023-09-07 17:46:05,627 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:46:05.627092 [2023-09-07 17:46:05,631 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:46:05,631 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:46:07,441 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:46:07,441 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain free-basic-rosy-SKALE - Process is running: PID = 30622 [2023-09-07 17:46:07,454 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:46:07,455 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:46:07,456 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:46:07,456 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:46:07,457 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:46:07,460 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:46:07,489 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:46:07,672 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:46:07,895 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:46:07,992 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:46:07,993 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2537 [2023-09-07 17:46:07,993 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:46:07,994 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:46:07,994 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:46:07,994 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:46:07,995 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:46:08,000 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:46:08,002 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:46:08.002657 [2023-09-07 17:46:08,140 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:46:08,171 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:46:08,221 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:46:08,222 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:46:08,231 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:46:08,231 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694108768.169845, 'te': 1694108768.1857913, 'initial_status': True} ====================================================================================================  [2023-09-07 17:46:08,233 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:46:08.233300 [2023-09-07 17:46:08,239 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:46:08,240 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:46:10,175 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108760, schain_finish_ts: 1692377364 [2023-09-07 17:46:10,595 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108760, schain_finish_ts: 1693742004 [2023-09-07 17:46:11,790 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:46:11,790 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain submissive-well-off-rotanev - Process is running: PID = 24912 [2023-09-07 17:46:11,791 INFO][1][MainProcess][MainThread] - core.schains.process_manager:88 - Process manager procedure finished [2023-09-07 17:46:11,791 INFO][1][MainProcess][MainThread] - __main__:65 - Sleeping for 90s after run_process_manager [2023-09-07 17:46:16,634 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108760, schain_finish_ts: 1692637668 [2023-09-07 17:46:23,067 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:46:23,068 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:46:23,068 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:46:23,068 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:46:23,068 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:46:23,068 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 17:46:23,090 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:46:23,112 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:46:23,113 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:46:23,115 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:46:23.115445 [2023-09-07 17:46:23,124 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:46:23,126 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:46:23,135 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:46:23,141 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694108783.1212287, 'te': 1694108783.1414814, 'initial_status': True} ====================================================================================================  [2023-09-07 17:46:23,144 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:46:23.144633 [2023-09-07 17:46:23,152 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:46:23,153 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:46:23,153 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:46:23,153 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:46:23,154 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:46:23,154 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:46:23,154 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:46:23,154 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:46:23,158 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:46:23,161 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:46:23,187 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:46:23,310 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:46:23,485 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:46:23,653 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:46:23,791 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:46:23,792 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2530 [2023-09-07 17:46:23,793 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:46:23,793 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:46:23,793 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:46:23,794 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:46:23,794 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:46:23,801 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:46:23,803 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:46:23.803488 [2023-09-07 17:46:23,819 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:46:24,028 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:46:24,029 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2517 [2023-09-07 17:46:24,030 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:46:24,031 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:46:24,031 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:46:24,031 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:46:24,032 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:46:24,042 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:46:24,045 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:46:24.045527 [2023-09-07 17:46:24,161 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:46:24,188 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:46:24,217 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:46:24,219 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:46:24,226 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:46:24,226 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694108784.1868005, 'te': 1694108784.193046, 'initial_status': True} ====================================================================================================  [2023-09-07 17:46:24,228 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:46:24.228497 [2023-09-07 17:46:24,232 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:46:24,233 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:46:24,406 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:46:24,430 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:46:24,457 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:46:24,459 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:46:24,465 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:46:24,466 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694108784.4290605, 'te': 1694108784.4348555, 'initial_status': True} ====================================================================================================  [2023-09-07 17:46:24,467 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:46:24.467700 [2023-09-07 17:46:24,472 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:46:24,472 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:46:26,578 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:46:26,578 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:46:26,579 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:46:26,579 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:46:26,579 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:46:26,579 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:46:26,581 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:46:26,584 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 17:46:26,607 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:46:26,707 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:46:26,792 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:46:27,009 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:46:27,105 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:46:27,106 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2533 [2023-09-07 17:46:27,107 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:46:27,107 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:46:27,108 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:46:27,108 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:46:27,108 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:46:27,114 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:46:27,116 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:46:27.116336 [2023-09-07 17:46:27,249 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:46:27,273 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:46:27,301 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:46:27,303 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:46:27,310 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:46:27,310 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694108787.2718067, 'te': 1694108787.279079, 'initial_status': True} ====================================================================================================  [2023-09-07 17:46:27,312 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:46:27.312394 [2023-09-07 17:46:27,317 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:46:27,317 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:46:31,854 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108760, schain_finish_ts: 0 [2023-09-07 17:46:32,023 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108760, schain_finish_ts: 1694108304 [2023-09-07 17:46:35,641 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108760, schain_finish_ts: 1693742304 [2023-09-07 17:46:36,834 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:46:36,834 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:46:36,836 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:46:36.836748 [2023-09-07 17:46:36,844 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:46:36,844 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:46:36,851 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:46:36,857 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694108796.8427062, 'te': 1694108796.8573043, 'initial_status': True} ====================================================================================================  [2023-09-07 17:46:36,859 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:46:36.859893 [2023-09-07 17:46:36,866 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:46:36,866 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:46:37,964 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108760, schain_finish_ts: 1693412520 [2023-09-07 17:46:41,866 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:46:41,867 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:46:41,867 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 17:46:41,906 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:46:41,906 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:46:41,907 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:46:42,148 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:46:42,148 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:46:42,149 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:46:42,149 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:46:42,149 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:46:42,150 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:46:42,151 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:46:42,152 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:46:42,182 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:46:42,367 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:46:42,492 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:46:42,589 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:46:42,591 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2534 [2023-09-07 17:46:42,592 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:46:42,592 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:46:42,592 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:46:42,593 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:46:42,593 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:46:42,602 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:46:42,605 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:46:42.605213 [2023-09-07 17:46:42,821 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:46:42,853 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:46:42,884 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:46:42,887 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:46:42,893 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:46:42,894 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694108802.8507836, 'te': 1694108802.8588147, 'initial_status': True} ====================================================================================================  [2023-09-07 17:46:42,895 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:46:42.895931 [2023-09-07 17:46:42,901 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:46:42,902 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:46:45,280 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:46:45,280 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:46:45,283 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:46:45.283737 [2023-09-07 17:46:45,294 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:46:45,294 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:46:45,301 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:46:45,306 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694108805.291116, 'te': 1694108805.305804, 'initial_status': True} ====================================================================================================  [2023-09-07 17:46:45,307 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:46:45.307619 [2023-09-07 17:46:45,312 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:46:45,312 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:46:54,614 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108808, schain_finish_ts: 1693742004 [2023-09-07 17:46:58,709 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:46:58,710 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:46:58,710 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:46:58,710 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:46:58,710 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:46:58,711 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:46:58,732 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:46:58,927 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:46:59,073 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:46:59,074 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:46:59,076 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:46:59.076196 [2023-09-07 17:46:59,077 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:46:59,084 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:46:59,084 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:46:59,091 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:46:59,096 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694108819.0822928, 'te': 1694108819.0958085, 'initial_status': True} ====================================================================================================  [2023-09-07 17:46:59,098 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:46:59.098882 [2023-09-07 17:46:59,103 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:46:59,104 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:46:59,189 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:46:59,190 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2538 [2023-09-07 17:46:59,190 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:46:59,190 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:46:59,191 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:46:59,191 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:46:59,191 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:46:59,197 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:46:59,199 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:46:59.199302 [2023-09-07 17:46:59,342 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:46:59,372 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:46:59,407 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:46:59,410 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:46:59,421 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:46:59,422 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694108819.3703823, 'te': 1694108819.3773735, 'initial_status': True} ====================================================================================================  [2023-09-07 17:46:59,426 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:46:59.426354 [2023-09-07 17:46:59,432 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:46:59,434 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:47:01,641 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108820, schain_finish_ts: 1692637668 [2023-09-07 17:47:06,071 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:47:08,607 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:47:08,607 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:47:08,608 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:47:08,608 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:47:08,608 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:47:08,609 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:47:08,646 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:47:08,903 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:09,051 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:09,172 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:47:09,173 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2531 [2023-09-07 17:47:09,174 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:47:09,174 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:47:09,174 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:47:09,175 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:47:09,175 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:47:09,184 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:47:09,186 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:47:09.186552 [2023-09-07 17:47:09,347 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:09,375 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:47:09,410 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:47:09,413 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:47:09,423 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:47:09,426 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694108829.3737319, 'te': 1694108829.3827925, 'initial_status': True} ====================================================================================================  [2023-09-07 17:47:09,428 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:47:09.428435 [2023-09-07 17:47:09,433 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:47:09,433 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:47:11,232 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108820, schain_finish_ts: 1692377364 [2023-09-07 17:47:14,729 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108832, schain_finish_ts: 0 [2023-09-07 17:47:15,309 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:47:15,310 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:47:15,310 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:47:15,310 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:47:15,311 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:47:15,312 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:47:15,314 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:47:15,317 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 17:47:15,372 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:47:15,612 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:15,641 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108832, schain_finish_ts: 1693412520 [2023-09-07 17:47:15,754 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:15,916 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:47:15,919 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2534 [2023-09-07 17:47:15,921 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:47:15,921 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:47:15,921 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:47:15,922 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:47:15,923 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:47:15,936 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:47:15,940 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:47:15.940114 [2023-09-07 17:47:16,132 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:16,163 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:47:16,203 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:47:16,206 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:47:16,215 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:47:16,215 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694108836.1604934, 'te': 1694108836.1693153, 'initial_status': True} ====================================================================================================  [2023-09-07 17:47:16,217 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:47:16.217420 [2023-09-07 17:47:16,222 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:47:16,223 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:47:22,221 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:47:22,221 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:47:22,222 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 17:47:22,572 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:47:22,573 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:47:22,573 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:47:22,574 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:47:22,575 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:47:22,576 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:47:22,579 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:47:22,580 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 17:47:22,609 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:47:22,810 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:22,965 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:23,064 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:47:23,065 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2518 [2023-09-07 17:47:23,066 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:47:23,066 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:47:23,066 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:47:23,067 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:47:23,068 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:47:23,078 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:47:23,081 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:47:23.081485 [2023-09-07 17:47:23,217 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:23,245 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:47:23,272 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:47:23,274 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:47:23,281 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:47:23,282 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694108843.2435813, 'te': 1694108843.2495027, 'initial_status': True} ====================================================================================================  [2023-09-07 17:47:23,283 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:47:23.283799 [2023-09-07 17:47:23,288 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:47:23,289 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:47:26,076 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:47:26,077 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:47:26,077 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:47:26,077 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:47:26,077 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:47:26,078 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:47:26,082 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:47:26,085 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:47:26,106 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:47:26,379 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:26,507 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:26,610 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:47:26,611 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2535 [2023-09-07 17:47:26,612 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:47:26,612 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:47:26,612 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:47:26,613 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:47:26,613 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:47:26,619 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:47:26,620 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:47:26.620898 [2023-09-07 17:47:26,754 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:26,780 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:47:26,813 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:47:26,815 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:47:26,822 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:47:26,822 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694108846.778721, 'te': 1694108846.7841694, 'initial_status': True} ====================================================================================================  [2023-09-07 17:47:26,824 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:47:26.824127 [2023-09-07 17:47:26,828 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:47:26,829 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:47:28,874 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108844, schain_finish_ts: 1694108304 [2023-09-07 17:47:29,661 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:47:29,661 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:47:29,665 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:47:29.665175 [2023-09-07 17:47:29,673 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:47:29,674 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:47:29,683 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:47:29,688 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694108849.6722615, 'te': 1694108849.6885476, 'initial_status': True} ====================================================================================================  [2023-09-07 17:47:29,691 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:47:29.691174 [2023-09-07 17:47:29,695 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:47:29,696 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:47:30,159 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:47:30,159 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:47:30,162 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:47:30.162794 [2023-09-07 17:47:30,171 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:47:30,171 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:47:30,179 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:47:30,188 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694108850.16938, 'te': 1694108850.1877565, 'initial_status': True} ====================================================================================================  [2023-09-07 17:47:30,190 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:47:30.190740 [2023-09-07 17:47:30,195 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:47:30,195 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:47:30,949 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108844, schain_finish_ts: 1693742304 [2023-09-07 17:47:34,403 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108844, schain_finish_ts: 1693742004 [2023-09-07 17:47:35,665 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:47:35,665 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:47:35,669 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 17:47:36,312 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:47:36,312 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:47:36,317 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:47:36.317676 [2023-09-07 17:47:36,329 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:47:36,329 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:47:36,336 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:47:36,341 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694108856.3260267, 'te': 1694108856.3412316, 'initial_status': True} ====================================================================================================  [2023-09-07 17:47:36,343 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:47:36.343230 [2023-09-07 17:47:36,347 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:47:36,348 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:47:38,860 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:47:38,860 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:47:38,861 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:47:38,861 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:47:38,862 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:47:38,863 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:47:38,883 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:47:39,091 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:39,374 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108856, schain_finish_ts: 1692637668 [2023-09-07 17:47:39,432 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:39,556 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:47:39,558 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2539 [2023-09-07 17:47:39,561 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:47:39,561 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:47:39,562 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:47:39,563 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:47:39,564 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:47:39,578 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:47:39,590 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:47:39.590624 [2023-09-07 17:47:39,814 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:39,844 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:47:39,894 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:47:39,898 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:47:39,905 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:47:39,906 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694108859.8416452, 'te': 1694108859.8494053, 'initial_status': True} ====================================================================================================  [2023-09-07 17:47:39,908 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:47:39.908073 [2023-09-07 17:47:39,912 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:47:39,912 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:47:41,884 INFO][1][MainProcess][MainThread] - core.schains.cleaner:65 - Cleaner process started [2023-09-07 17:47:41,909 INFO][30846][cleaner][MainThread] - core.schains.cleaner:110 - Cleaner procedure started. [2023-09-07 17:47:42,853 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:47:42,854 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:47:42,854 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:47:42,854 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:47:42,854 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:47:42,855 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:47:42,874 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:47:43,073 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:43,218 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:43,344 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:47:43,345 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2532 [2023-09-07 17:47:43,346 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:47:43,346 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:47:43,347 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:47:43,347 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:47:43,347 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:47:43,353 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:47:43,355 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:47:43.355815 [2023-09-07 17:47:43,599 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:43,627 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:47:43,654 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:47:43,656 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:47:43,663 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:47:43,664 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694108863.6255429, 'te': 1694108863.632092, 'initial_status': True} ====================================================================================================  [2023-09-07 17:47:43,665 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:47:43.665573 [2023-09-07 17:47:43,669 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:47:43,670 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:47:44,710 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:47:47,732 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:47:47,733 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:47:47,733 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:47:47,733 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:47:47,734 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:47:47,734 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:47:47,736 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:47:47,736 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 17:47:47,765 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:47:48,043 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:48,316 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:48,464 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:47:48,464 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2535 [2023-09-07 17:47:48,465 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:47:48,466 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:47:48,466 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:47:48,466 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:47:48,467 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:47:48,474 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:47:48,476 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:47:48.476047 [2023-09-07 17:47:48,685 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:48,714 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:47:48,758 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:47:48,760 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 17:47:48,761 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:47:48,771 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:47:48,772 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694108868.712119, 'te': 1694108868.7209735, 'initial_status': True} ====================================================================================================  [2023-09-07 17:47:48,774 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:47:48.774476 [2023-09-07 17:47:48,781 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:47:48,782 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:47:48,947 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:47:48,948 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:47:48,948 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:47:48,948 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:47:49,139 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:47:49,140 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:47:49,142 INFO][8][curly-red-alterf][T_1] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 17:47:49,147 ERROR][8][curly-red-alterf][T_1] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:47:49,148 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:47:52,730 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:47:52,730 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:47:52,732 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:47:52.732670 [2023-09-07 17:47:52,740 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:47:52,741 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:47:52,747 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:47:52,752 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694108872.7392843, 'te': 1694108872.7526739, 'initial_status': True} ====================================================================================================  [2023-09-07 17:47:52,755 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:47:52.755462 [2023-09-07 17:47:52,761 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:47:52,761 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:47:55,073 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108868, schain_finish_ts: 1692377364 [2023-09-07 17:47:55,109 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 17:47:55,109 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 17:47:55,255 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108868, schain_finish_ts: 1693412520 [2023-09-07 17:47:55,613 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108868, schain_finish_ts: 0 [2023-09-07 17:47:56,375 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:47:56,375 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:47:56,375 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:47:56,376 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:47:56,376 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:47:56,376 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:47:56,379 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:47:56,379 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 17:47:56,409 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:47:56,606 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:56,782 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:56,917 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:47:56,922 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2536 [2023-09-07 17:47:56,923 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:47:56,923 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:47:56,923 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:47:56,924 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:47:56,924 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:47:56,932 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:47:56,935 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:47:56.935471 [2023-09-07 17:47:57,325 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:57,358 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:47:57,392 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:47:57,395 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:47:57,406 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:47:57,406 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694108877.3567865, 'te': 1694108877.3640544, 'initial_status': True} ====================================================================================================  [2023-09-07 17:47:57,408 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:47:57.408394 [2023-09-07 17:47:57,413 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:47:57,414 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:47:57,442 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:47:57,442 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:47:57,442 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:47:57,443 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:47:57,443 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:47:57,443 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:47:57,443 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 17:47:57,443 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:47:57,443 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:47:57,445 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:47:57,447 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 17:47:57,474 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:47:57,625 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108868, schain_finish_ts: 1694108304 [2023-09-07 17:47:57,722 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:57,906 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:58,023 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:47:58,024 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2519 [2023-09-07 17:47:58,024 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:47:58,025 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:47:58,025 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:47:58,025 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:47:58,026 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:47:58,033 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:47:58,035 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:47:58.035533 [2023-09-07 17:47:58,353 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:47:58,353 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:47:58,356 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:47:58.356311 [2023-09-07 17:47:58,365 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:47:58,365 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:47:58,369 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:47:58,371 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:47:58,378 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694108878.3636413, 'te': 1694108878.3782203, 'initial_status': True} ====================================================================================================  [2023-09-07 17:47:58,381 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:47:58.381430 [2023-09-07 17:47:58,387 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:47:58,387 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:47:58,395 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:47:58,428 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:47:58,430 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:47:58,437 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:47:58,437 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694108878.3937004, 'te': 1694108878.3995986, 'initial_status': True} ====================================================================================================  [2023-09-07 17:47:58,440 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:47:58.440246 [2023-09-07 17:47:58,444 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:47:58,444 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:47:58,772 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:47:58,772 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:47:58,772 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 17:47:59,278 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:47:59,279 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:47:59,281 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:47:59.281097 [2023-09-07 17:47:59,291 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:47:59,291 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:47:59,299 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:47:59,304 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694108879.289966, 'te': 1694108879.3038785, 'initial_status': True} ====================================================================================================  [2023-09-07 17:47:59,305 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:47:59.305802 [2023-09-07 17:47:59,310 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:47:59,310 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:47:59,579 INFO][30846][cleaner][MainThread] - core.schains.cleaner:116 - sChains on contracts: ['portly-passionate-sirius', 'curly-red-alterf', 'howling-hoedus', 'green-giddy-denebola', 'wan-tasty-kitalpha', 'free-basic-rosy-SKALE', 'submissive-well-off-rotanev'] sChains on node: ['curly-red-alterf', 'free-basic-rosy-SKALE', 'green-giddy-denebola', 'howling-hoedus', 'portly-passionate-sirius', 'submissive-well-off-rotanev', 'wan-tasty-kitalpha'] [2023-09-07 17:47:59,580 INFO][30846][cleaner][MainThread] - core.schains.cleaner:132 - Cleanup procedure finished [2023-09-07 17:47:59,586 INFO][1][MainProcess][MainThread] - core.schains.cleaner:67 - Cleaner process is joined. [2023-09-07 17:47:59,586 INFO][1][MainProcess][MainThread] - core.schains.cleaner:68 - Terminating the process [2023-09-07 17:47:59,586 INFO][1][MainProcess][MainThread] - __main__:70 - Sleeping for 90s after run_cleaner [2023-09-07 17:48:01,834 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108868, schain_finish_ts: 1693742304 [2023-09-07 17:48:02,988 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:48:02,988 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:48:02,989 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:48:02,989 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:48:02,989 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:48:02,989 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:48:03,012 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:48:03,348 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:48:03,538 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:48:03,725 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:48:03,726 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2540 [2023-09-07 17:48:03,727 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:48:03,728 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:48:03,728 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:48:03,728 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:48:03,729 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:48:03,744 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:48:03,748 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:48:03.748373 [2023-09-07 17:48:03,906 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:48:03,937 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:48:03,969 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:48:03,971 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:48:03,979 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:48:03,979 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694108883.9357772, 'te': 1694108883.9455452, 'initial_status': True} ====================================================================================================  [2023-09-07 17:48:03,981 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:48:03.981058 [2023-09-07 17:48:03,985 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:48:03,986 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:48:11,669 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:40 - Collecting rotation history for submissive-well-off-rotanev... [2023-09-07 17:48:12,800 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:50 - Rotation data for submissive-well-off-rotanev: Rotation(leaving_node_id=65, new_node_id=65, freeze_until=1694108100, rotation_counter=8) [2023-09-07 17:48:15,662 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:48:15,795 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108868, schain_finish_ts: 1693742004 [2023-09-07 17:48:16,113 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 17:48:16,288 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:48:16,289 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:48:16,289 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:48:16,289 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:48:16,454 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:48:16,454 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:48:16,456 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 17:48:16,461 ERROR][11][wan-tasty-kitalpha][T_1] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:48:16,461 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:48:16,934 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:48:16,934 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:48:16,934 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:48:16,934 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 17:48:16,934 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 17:48:16,935 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:48:16,936 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:48:16,936 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 17:48:16,961 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:48:17,178 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:48:17,343 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:48:17,457 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:48:17,458 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2533 [2023-09-07 17:48:17,459 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:48:17,459 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:48:17,460 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:48:17,460 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:48:17,460 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:48:17,467 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:48:17,469 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:48:17.469439 [2023-09-07 17:48:17,683 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108892, schain_finish_ts: 1692637668 [2023-09-07 17:48:17,694 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:48:17,735 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:48:17,780 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:48:17,784 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:48:17,796 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:48:17,797 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694108897.7341597, 'te': 1694108897.7406003, 'initial_status': True} ====================================================================================================  [2023-09-07 17:48:17,799 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:48:17.799333 [2023-09-07 17:48:17,804 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:48:17,805 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:48:18,787 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:48:18,788 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:48:18,792 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:48:18.792771 [2023-09-07 17:48:18,799 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:48:18,845 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:48:18,846 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:48:18,846 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:48:18,846 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:48:18,847 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:48:18,847 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:48:18,850 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:48:18,854 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:48:18,887 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:48:19,112 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:48:19,281 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:48:19,401 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108892, schain_finish_ts: 1692377364 [2023-09-07 17:48:19,408 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:48:19,409 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2536 [2023-09-07 17:48:19,411 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:48:19,411 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:48:19,411 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:48:19,412 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:48:19,412 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:48:19,423 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:48:19,431 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:48:19.431358 [2023-09-07 17:48:19,590 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:48:19,617 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:48:19,654 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:48:19,656 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:48:19,664 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:48:19,664 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694108899.6158583, 'te': 1694108899.6251109, 'initial_status': True} ====================================================================================================  [2023-09-07 17:48:19,666 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:48:19.666592 [2023-09-07 17:48:19,681 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:48:19,706 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:48:20,346 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108892, schain_finish_ts: 1693412520 [2023-09-07 17:48:20,598 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:48:20,598 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:48:20,599 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:48:20,599 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:48:20,599 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:48:20,600 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:48:20,602 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:48:20,606 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:48:20,635 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:48:20,661 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:48:20,661 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:48:20,664 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:48:20.664114 [2023-09-07 17:48:20,673 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:48:20,673 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:48:20,679 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:48:20,687 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694108900.6722188, 'te': 1694108900.6872368, 'initial_status': True} ====================================================================================================  [2023-09-07 17:48:20,689 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:48:20.689113 [2023-09-07 17:48:20,692 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:48:20,693 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:48:20,849 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:48:21,003 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:48:21,131 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:48:21,134 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2520 [2023-09-07 17:48:21,135 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:48:21,135 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:48:21,136 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:48:21,137 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:48:21,140 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:48:21,150 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:48:21,153 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:48:21.153286 [2023-09-07 17:48:21,378 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:48:21,411 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:48:21,453 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:48:21,455 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:48:21,463 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:48:21,464 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694108901.4091966, 'te': 1694108901.4166596, 'initial_status': True} ====================================================================================================  [2023-09-07 17:48:21,468 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:48:21.468610 [2023-09-07 17:48:21,476 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:48:21,478 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:48:21,482 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:48:21,482 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:48:21,482 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:48:21,482 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:48:21,483 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:48:21,483 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:48:21,485 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:48:21,487 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:48:21,529 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:48:21,871 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:48:22,070 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:48:22,221 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:48:22,222 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2537 [2023-09-07 17:48:22,223 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:48:22,223 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:48:22,223 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:48:22,224 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:48:22,224 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:48:22,231 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:48:22,237 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:48:22.237271 [2023-09-07 17:48:22,396 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:48:22,423 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:48:22,441 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:48:22,441 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:48:22,443 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:48:22.443973 [2023-09-07 17:48:22,450 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:48:22,450 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:48:22,456 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:48:22,461 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694108902.4490244, 'te': 1694108902.4608593, 'initial_status': True} ====================================================================================================  [2023-09-07 17:48:22,463 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:48:22.463160 [2023-09-07 17:48:22,469 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:48:22,469 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:48:22,473 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:48:22,475 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:48:22,482 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:48:22,483 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694108902.4217877, 'te': 1694108902.4285035, 'initial_status': True} ====================================================================================================  [2023-09-07 17:48:22,484 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:48:22.484873 [2023-09-07 17:48:22,490 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:48:22,490 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:48:23,274 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:48:23,274 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:48:23,276 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:48:23.276741 [2023-09-07 17:48:23,286 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:48:23,286 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:48:23,294 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:48:23,299 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694108903.2837527, 'te': 1694108903.2994163, 'initial_status': True} ====================================================================================================  [2023-09-07 17:48:23,302 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:48:23.302135 [2023-09-07 17:48:23,307 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:48:23,307 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:48:24,710 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108892, schain_finish_ts: 1694108304 [2023-09-07 17:48:25,869 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:48:25,869 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:48:25,869 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:48:27,373 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108904, schain_finish_ts: 0 [2023-09-07 17:48:28,069 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 63 -> 65 [2023-09-07 17:48:28,483 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:48:28,483 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:48:28,483 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 17:48:31,947 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108904, schain_finish_ts: 1693742304 [2023-09-07 17:48:32,873 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 23 -> 24 [2023-09-07 17:48:33,082 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:48:33,082 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:48:33,082 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:48:33,082 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 17:48:33,083 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 17:48:33,083 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:48:33,084 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:48:33,085 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 17:48:33,114 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:48:33,368 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:48:33,692 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:48:33,828 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:48:33,829 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2541 [2023-09-07 17:48:33,831 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:48:33,831 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:48:33,831 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:48:33,832 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:48:33,832 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:48:33,843 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:48:33,846 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:48:33.846216 [2023-09-07 17:48:34,015 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:48:34,067 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:48:34,101 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:48:34,103 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:48:34,110 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:48:34,111 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694108914.0655158, 'te': 1694108914.07319, 'initial_status': True} ====================================================================================================  [2023-09-07 17:48:34,112 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:48:34.112671 [2023-09-07 17:48:34,116 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:48:34,117 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:48:34,984 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:48:34,984 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:48:34,986 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:48:34.986827 [2023-09-07 17:48:34,992 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:48:37,324 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 9 -> 64 [2023-09-07 17:48:38,152 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 17:48:38,152 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 17:48:42,205 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 35 -> 16 [2023-09-07 17:48:42,884 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108916, schain_finish_ts: 1693742004 [2023-09-07 17:48:44,028 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:48:44,028 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:48:44,028 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:48:44,028 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:48:44,029 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:48:44,029 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:48:44,076 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:48:44,379 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:48:44,645 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:48:44,790 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:48:44,794 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2534 [2023-09-07 17:48:44,795 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:48:44,795 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:48:44,795 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:48:44,796 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:48:44,796 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:48:44,802 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:48:44,804 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:48:44.804712 [2023-09-07 17:48:44,953 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:48:44,987 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:48:45,018 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:48:45,020 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:48:45,027 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:48:45,027 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694108924.9863694, 'te': 1694108924.9921398, 'initial_status': True} ====================================================================================================  [2023-09-07 17:48:45,029 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:48:45.029275 [2023-09-07 17:48:45,035 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:48:45,036 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:48:46,662 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:48:47,468 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 55 -> 63 [2023-09-07 17:48:49,408 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108928, schain_finish_ts: 0 [2023-09-07 17:48:50,538 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:48:50,538 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:48:50,538 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:48:51,830 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108928, schain_finish_ts: 1692637668 [2023-09-07 17:48:52,964 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:48:52,964 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:48:52,964 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:48:52,964 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:48:52,965 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:48:52,965 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:48:52,967 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:48:52,969 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 17:48:52,997 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:48:53,214 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 14 -> 11 [2023-09-07 17:48:53,373 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:48:53,555 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:48:53,666 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:48:53,667 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2537 [2023-09-07 17:48:53,667 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:48:53,668 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:48:53,668 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:48:53,668 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:48:53,669 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:48:53,675 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:48:53,677 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:48:53.677859 [2023-09-07 17:48:53,851 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:48:53,881 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:48:53,910 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:48:53,912 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:48:53,924 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:48:53,925 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694108933.8789163, 'te': 1694108933.887527, 'initial_status': True} ====================================================================================================  [2023-09-07 17:48:53,927 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:48:53.927261 [2023-09-07 17:48:53,931 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:48:53,932 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:48:54,165 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 17:48:54,165 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 17:48:54,835 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:48:54,835 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:48:54,837 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:48:54.837685 [2023-09-07 17:48:54,845 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:48:54,845 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:48:54,850 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:48:54,855 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694108934.84396, 'te': 1694108934.8549068, 'initial_status': True} ====================================================================================================  [2023-09-07 17:48:54,856 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:48:54.856741 [2023-09-07 17:48:54,860 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:48:54,860 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:48:57,680 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 17:48:57,847 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:48:57,847 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:48:57,847 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:48:57,848 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:48:58,029 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:48:58,029 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:48:58,032 INFO][8][curly-red-alterf][T_0] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 17:48:58,036 ERROR][8][curly-red-alterf][T_0] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:48:58,037 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:48:59,264 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 56 -> 60 [2023-09-07 17:48:59,429 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108928, schain_finish_ts: 1693412520 [2023-09-07 17:49:00,567 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108928, schain_finish_ts: 1692377364 [2023-09-07 17:49:00,575 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:49:00,575 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:49:00,575 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:49:00,576 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:49:00,576 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:49:00,576 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:49:00,576 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:49:00,577 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:49:00,617 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:49:00,965 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:01,081 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108940, schain_finish_ts: 1693742304 [2023-09-07 17:49:01,319 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:01,457 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:49:01,458 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2538 [2023-09-07 17:49:01,459 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:49:01,459 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:49:01,460 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:49:01,460 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:49:01,460 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:49:01,466 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:49:01,468 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:49:01.468344 [2023-09-07 17:49:01,637 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:01,671 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:49:01,699 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:49:01,700 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:49:01,708 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:49:01,708 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694108941.669896, 'te': 1694108941.6767, 'initial_status': True} ====================================================================================================  [2023-09-07 17:49:01,709 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:49:01.709888 [2023-09-07 17:49:01,716 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:49:01,718 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:49:01,724 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:49:01,725 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:49:01,725 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:49:01,725 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:49:01,725 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:49:01,726 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:49:01,727 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:49:01,732 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 17:49:01,755 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:49:01,973 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:02,180 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:02,248 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:49:02,248 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:49:02,249 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:49:02,249 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:49:02,249 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:49:02,249 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 17:49:02,286 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:49:02,295 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:49:02,299 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2521 [2023-09-07 17:49:02,303 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:49:02,303 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:49:02,303 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:49:02,303 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:49:02,304 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:49:02,402 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:49:02,402 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:49:02,404 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:49:02.404791 [2023-09-07 17:49:02,412 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:49:02,412 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:49:02,419 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:49:02,423 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694108942.4105542, 'te': 1694108942.423408, 'initial_status': True} ====================================================================================================  [2023-09-07 17:49:02,424 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:49:02,425 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:49:02.425670 [2023-09-07 17:49:02,431 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:49:02,431 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:49:02,436 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:49:02.436132 [2023-09-07 17:49:02,527 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:02,660 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:02,685 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:49:02,720 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:49:02,724 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:49:02,732 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:49:02,732 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694108942.6835358, 'te': 1694108942.6893744, 'initial_status': True} ====================================================================================================  [2023-09-07 17:49:02,734 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:49:02.734316 [2023-09-07 17:49:02,739 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:49:02,740 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:49:02,850 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:02,971 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:49:02,972 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2542 [2023-09-07 17:49:02,973 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:49:02,973 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:49:02,973 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:49:02,974 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:49:02,974 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:49:02,981 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:49:02,983 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:49:02.983321 [2023-09-07 17:49:03,158 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:03,187 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:49:03,216 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:49:03,218 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:49:03,233 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:49:03,234 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694108943.1862538, 'te': 1694108943.1942244, 'initial_status': True} ====================================================================================================  [2023-09-07 17:49:03,236 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:49:03.236028 [2023-09-07 17:49:03,246 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:49:03,247 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:49:03,506 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:49:03,506 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:49:03,508 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:49:03.508796 [2023-09-07 17:49:03,515 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:49:03,515 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:49:03,521 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:49:03,526 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694108943.5138264, 'te': 1694108943.5262334, 'initial_status': True} ====================================================================================================  [2023-09-07 17:49:03,528 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:49:03.528123 [2023-09-07 17:49:03,532 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:49:03,533 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:49:03,824 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108940, schain_finish_ts: 1694108304 [2023-09-07 17:49:04,972 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:49:04,973 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:49:04,973 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:49:05,385 ERROR][24912][submissive-well-off-rotanev][T_0] - core.schains.task:26 - Task submissive-well-off-rotanev-config failed with max() arg is an empty sequence Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 63, in execute self.am.upstream_config() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 194, in upstream_config new_config = create_new_upstream_config( File "/usr/src/admin/core/schains/config/main.py", line 51, in create_new_upstream_config schain_config = generate_schain_config_with_skale( File "/usr/src/admin/core/schains/config/generator.py", line 213, in generate_schain_config_with_skale node_groups = get_previous_schain_groups(skale, schain_name) File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 56, in get_previous_schain_groups _add_previous_schain_rotations_state( File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 119, in _add_previous_schain_rotations_state new_node_id = max(previous_nodes.items(), key=lambda x: x[1]['finish_ts'])[0] ValueError: max() arg is an empty sequence [2023-09-07 17:49:05,386 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:49:13,458 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 17:49:13,637 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:49:13,637 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:49:13,637 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:49:13,638 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:49:13,819 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:49:13,819 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:49:13,821 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 17:49:13,829 ERROR][11][wan-tasty-kitalpha][T_0] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:49:13,830 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:49:16,977 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108952, schain_finish_ts: 1693742004 [2023-09-07 17:49:17,596 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:49:18,130 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:49:18,130 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:49:18,131 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:49:18,131 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 17:49:18,132 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 17:49:18,132 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:49:18,136 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:49:18,136 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 17:49:18,167 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:49:18,401 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:18,571 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:18,712 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:49:18,714 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2535 [2023-09-07 17:49:18,714 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:49:18,715 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:49:18,715 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:49:18,715 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:49:18,716 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:49:18,728 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:49:18,734 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:49:18.734281 [2023-09-07 17:49:19,017 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:19,059 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:49:19,093 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:49:19,095 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:49:19,104 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:49:19,105 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694108959.0558424, 'te': 1694108959.064398, 'initial_status': True} ====================================================================================================  [2023-09-07 17:49:19,106 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:49:19.106868 [2023-09-07 17:49:19,111 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:49:19,112 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:49:19,938 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:49:19,939 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:49:19,942 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:49:19.942510 [2023-09-07 17:49:19,947 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:49:28,452 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108952, schain_finish_ts: 0 [2023-09-07 17:49:29,516 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108952, schain_finish_ts: 1693412520 [2023-09-07 17:49:29,596 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:49:29,596 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:49:29,597 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:40 - Task submissive-well-off-rotanev-config finished with None [2023-09-07 17:49:29,601 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:42 - Running task submissive-well-off-rotanev-config [2023-09-07 17:49:29,602 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 17:49:29,646 INFO][1][MainProcess][MainThread] - core.schains.process_manager:57 - Process manager started [2023-09-07 17:49:29,870 INFO][1][MainProcess][MainThread] - core.schains.notifications:35 - Node account has 406903463189434189 WEI [2023-09-07 17:49:29,871 INFO][1][MainProcess][MainThread] - tools.notifications.messages:188 - Saving new balance state 3015 1 [2023-09-07 17:49:29,872 INFO][1][MainProcess][MainThread] - core.schains.process_manager:95 - Fetching schains to monitor... [2023-09-07 17:49:29,902 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108952, schain_finish_ts: 1692637668 [2023-09-07 17:49:30,683 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:49:30,683 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:49:30,683 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:49:30,685 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:49:30,685 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:49:30,685 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:49:30,687 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:49:30,692 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 17:49:30,719 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:49:30,930 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:31,074 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:49:31,075 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:49:31,075 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:49:31,075 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:49:31,076 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:49:31,076 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:49:31,077 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:49:31,078 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:49:31,108 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:49:31,117 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:31,274 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:49:31,275 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2539 [2023-09-07 17:49:31,276 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:49:31,276 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:49:31,277 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:49:31,277 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:49:31,277 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:49:31,286 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:49:31,288 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:49:31.288629 [2023-09-07 17:49:31,350 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': False} [2023-09-07 17:49:31,350 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:49:31,352 INFO][24912][submissive-well-off-rotanev][T_0] - web.models.schain:152 - Changing monitor_last_seen for submissive-well-off-rotanev to 2023-09-07 17:49:31.352641 [2023-09-07 17:49:31,357 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.action:190 - Creating new upstream_config rotation_id: 8, stream: regression-2.2 [2023-09-07 17:49:31,358 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:31,359 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.config.main:49 - Generating sChain config for submissive-well-off-rotanev [2023-09-07 17:49:31,542 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:31,570 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:49:31,602 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:49:31,607 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:49:31,618 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:49:31,621 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694108971.5689743, 'te': 1694108971.5767303, 'initial_status': True} ====================================================================================================  [2023-09-07 17:49:31,623 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:49:31.623825 [2023-09-07 17:49:31,629 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:49:31,630 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:49:31,737 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:31,857 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:49:31,860 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2538 [2023-09-07 17:49:31,861 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:49:31,863 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:49:31,864 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:49:31,864 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:49:31,864 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:49:31,872 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:49:31,874 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:49:31.874183 [2023-09-07 17:49:32,104 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:32,133 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:49:32,168 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:49:32,171 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:49:32,187 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:49:32,187 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694108972.1295369, 'te': 1694108972.1409724, 'initial_status': True} ====================================================================================================  [2023-09-07 17:49:32,189 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:49:32.189736 [2023-09-07 17:49:32,196 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:49:32,197 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:49:32,469 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:49:32,470 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:49:32,472 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:49:32.472661 [2023-09-07 17:49:32,480 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:49:32,480 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:49:32,487 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:49:32,491 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694108972.4792786, 'te': 1694108972.4915185, 'initial_status': True} ====================================================================================================  [2023-09-07 17:49:32,493 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:49:32.493366 [2023-09-07 17:49:32,499 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:49:32,499 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:49:32,672 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108952, schain_finish_ts: 1692377364 [2023-09-07 17:49:32,906 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:49:32,906 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:49:32,908 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:49:32.908681 [2023-09-07 17:49:32,916 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:49:32,917 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:49:32,925 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:49:32,930 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694108972.9149191, 'te': 1694108972.9302344, 'initial_status': True} ====================================================================================================  [2023-09-07 17:49:32,932 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:49:32.932145 [2023-09-07 17:49:32,936 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:49:32,937 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:49:33,814 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:49:33,815 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:49:33,815 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:49:33,815 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:49:33,815 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:49:33,816 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:49:33,822 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:49:33,824 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:49:33,860 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:49:34,106 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:34,303 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:34,434 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:49:34,435 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2522 [2023-09-07 17:49:34,436 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:49:34,436 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:49:34,436 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:49:34,437 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:49:34,437 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:49:34,448 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:49:34,451 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:49:34.451201 [2023-09-07 17:49:34,643 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:34,675 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:49:34,717 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:49:34,721 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:49:34,731 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:49:34,732 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694108974.6733623, 'te': 1694108974.683684, 'initial_status': True} ====================================================================================================  [2023-09-07 17:49:34,734 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:49:34.734774 [2023-09-07 17:49:34,739 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:49:34,740 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:49:35,534 INFO][1][MainProcess][MainThread] - core.schains.process_manager:109 - Get leaving_history for node ... [2023-09-07 17:49:35,628 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:49:35,628 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:49:35,631 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:49:35.631369 [2023-09-07 17:49:35,638 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:49:35,638 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:49:35,646 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:49:35,652 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694108975.6370127, 'te': 1694108975.6518142, 'initial_status': True} ====================================================================================================  [2023-09-07 17:49:35,653 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:49:35.653933 [2023-09-07 17:49:35,658 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:49:35,659 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:49:36,206 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108952, schain_finish_ts: 1693742304 [2023-09-07 17:49:37,353 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:49:37,353 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:49:37,355 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:49:37,358 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 17:49:37,359 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 17:49:37,359 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:49:37,362 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:49:37,367 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:49:37,400 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:49:37,506 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108976, schain_finish_ts: 1692377616 [2023-09-07 17:49:37,713 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:37,899 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:38,032 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:49:38,033 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2543 [2023-09-07 17:49:38,034 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:49:38,034 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:49:38,034 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:49:38,035 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:49:38,035 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:49:38,041 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:49:38,043 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:49:38.043397 [2023-09-07 17:49:38,227 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:38,254 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:49:38,284 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:49:38,287 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:49:38,296 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:49:38,297 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694108978.252441, 'te': 1694108978.2603083, 'initial_status': True} ====================================================================================================  [2023-09-07 17:49:38,298 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:49:38.298885 [2023-09-07 17:49:38,303 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:49:38,304 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:49:39,170 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 17:49:39,170 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 17:49:39,273 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:49:39,273 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:49:39,279 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:49:39.278944 [2023-09-07 17:49:39,286 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:49:39,520 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108976, schain_finish_ts: 0 [2023-09-07 17:49:39,893 INFO][1][MainProcess][MainThread] - core.schains.process_manager:117 - Got leaving sChains for the node: [] [2023-09-07 17:49:39,894 INFO][1][MainProcess][MainThread] - core.schains.process_manager:101 -  ==================================================================================================== Monitoring sChains Node ID: 16 sChains on node: [{'name': 'portly-passionate-sirius', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666281384, 'startBlock': 7803380, 'deposit': 0, 'index': 1, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x0000017723ca2', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'curly-red-alterf', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1676058936, 'startBlock': 8469646, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000001fe6cdfa', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'howling-hoedus', 'mainnetOwner': '0x1026467C91fdA40aCc7e6935d1E52f67274FD669', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666181400, 'startBlock': 7796660, 'deposit': 0, 'index': 0, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0xe5a600f4c2608', 'multitransactionMode': False, 'thresholdEncryption': False, 'active': True}, {'name': 'green-giddy-denebola', 'mainnetOwner': '0x8e835d5d9E28ffD259E8Fa8988Cacfc333A582B4', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674565164, 'startBlock': 8368049, 'deposit': 0, 'index': 4, 'generation': 1, 'originator': '0xd2001DAb6898127Be2F167B548691C87251D13C3', 'chainId': '0x00000585eb4b1', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'wan-tasty-kitalpha', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674568392, 'startBlock': 8368271, 'deposit': 0, 'index': 5, 'generation': 1, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x00000152e7be9', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'free-basic-rosy-SKALE', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1693996920, 'startBlock': 9646443, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000002bb90439', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'submissive-well-off-rotanev', 'mainnetOwner': '0x487360568fE5Ee4AE5c75782D5F63e9935B0E512', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1692361104, 'startBlock': 9539053, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000003791fa10', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}] Number of sChains on node: 7 Empty sChain structs: 0 ====================================================================================================  [2023-09-07 17:49:40,271 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:49:40,272 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain portly-passionate-sirius - Process is running: PID = 7 [2023-09-07 17:49:40,656 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:49:40,656 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain curly-red-alterf - Process is running: PID = 8 [2023-09-07 17:49:41,027 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:49:41,028 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain howling-hoedus - Process is running: PID = 9 [2023-09-07 17:49:41,406 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:49:41,406 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain green-giddy-denebola - Process is running: PID = 10 [2023-09-07 17:49:41,784 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:49:41,785 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain wan-tasty-kitalpha - Process is running: PID = 11 [2023-09-07 17:49:42,157 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:49:42,158 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain free-basic-rosy-SKALE - Process is running: PID = 30622 [2023-09-07 17:49:42,532 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:49:42,533 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain submissive-well-off-rotanev - Process is running: PID = 24912 [2023-09-07 17:49:42,533 INFO][1][MainProcess][MainThread] - core.schains.process_manager:88 - Process manager procedure finished [2023-09-07 17:49:42,533 INFO][1][MainProcess][MainThread] - __main__:65 - Sleeping for 90s after run_process_manager [2023-09-07 17:49:43,935 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108976, schain_finish_ts: 1694108304 [2023-09-07 17:49:44,088 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108976, schain_finish_ts: 1693742004 [2023-09-07 17:49:45,087 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:49:45,087 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:49:45,088 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 17:49:45,253 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:49:45,253 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:49:45,254 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:49:45,254 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:49:45,254 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:49:45,255 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:49:45,275 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:49:45,491 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:45,792 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:45,925 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:49:45,926 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2536 [2023-09-07 17:49:45,929 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:49:45,930 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:49:45,930 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:49:45,933 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:49:45,933 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:49:45,941 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:49:45,943 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:49:45.943706 [2023-09-07 17:49:46,135 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:49:46,162 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:49:46,194 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:49:46,197 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:49:46,207 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:49:46,208 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694108986.1608515, 'te': 1694108986.1686635, 'initial_status': True} ====================================================================================================  [2023-09-07 17:49:46,209 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:49:46.209593 [2023-09-07 17:49:46,214 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:49:46,214 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:49:48,544 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:49:58,431 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 17:49:58,431 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 17:49:58,604 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 17:49:58,799 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:49:58,799 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:49:58,799 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:49:58,800 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:49:58,974 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:49:58,974 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:49:58,976 INFO][8][curly-red-alterf][T_1] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 17:49:58,981 ERROR][8][curly-red-alterf][T_1] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:49:58,981 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:49:59,565 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108988, schain_finish_ts: 0 [2023-09-07 17:50:00,710 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:50:00,711 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:50:00,711 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:50:04,637 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108988, schain_finish_ts: 1693412520 [2023-09-07 17:50:05,795 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:50:05,795 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:50:05,796 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:50:05,796 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:50:05,796 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:50:05,796 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:50:05,798 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:50:05,798 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 17:50:05,830 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:50:06,113 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:06,398 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:06,548 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:50:06,549 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2540 [2023-09-07 17:50:06,550 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:50:06,550 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:50:06,551 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:50:06,551 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:50:06,551 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:50:06,557 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:50:06,558 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:50:06.558777 [2023-09-07 17:50:06,719 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:06,745 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:50:06,778 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:50:06,780 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:50:06,791 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:50:06,791 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109006.7435174, 'te': 1694109006.7525334, 'initial_status': True} ====================================================================================================  [2023-09-07 17:50:06,792 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:50:06.792879 [2023-09-07 17:50:06,798 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:50:06,799 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:50:07,031 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108988, schain_finish_ts: 1692637668 [2023-09-07 17:50:07,685 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:50:07,686 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:50:07,689 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:50:07.689502 [2023-09-07 17:50:07,696 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:50:07,697 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:50:07,703 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:50:07,708 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109007.6952355, 'te': 1694109007.7086895, 'initial_status': True} ====================================================================================================  [2023-09-07 17:50:07,710 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:50:07.710696 [2023-09-07 17:50:07,716 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:50:07,716 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:50:08,156 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:50:08,156 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:50:08,157 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:50:08,157 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:50:08,157 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:50:08,157 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:50:08,159 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:50:08,159 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:50:08,186 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:50:08,531 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:08,672 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:08,775 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:50:08,776 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2539 [2023-09-07 17:50:08,776 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:50:08,776 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:50:08,777 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:50:08,777 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:50:08,777 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:50:08,783 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:50:08,785 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:50:08.785707 [2023-09-07 17:50:08,943 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:08,971 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:50:09,000 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:50:09,002 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:50:09,010 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:50:09,011 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109008.969878, 'te': 1694109008.9768906, 'initial_status': True} ====================================================================================================  [2023-09-07 17:50:09,012 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:50:09.012649 [2023-09-07 17:50:09,018 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:50:09,018 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:50:09,777 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108988, schain_finish_ts: 1692377364 [2023-09-07 17:50:09,961 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:50:09,962 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:50:09,965 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:50:09.965007 [2023-09-07 17:50:09,976 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:50:09,977 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:50:09,983 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:50:09,990 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109009.973945, 'te': 1694109009.9902177, 'initial_status': True} ====================================================================================================  [2023-09-07 17:50:09,992 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:50:09.992806 [2023-09-07 17:50:09,999 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:50:10,000 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:50:10,355 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694108988, schain_finish_ts: 1693742304 [2023-09-07 17:50:10,940 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:50:10,940 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:50:10,941 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:50:10,941 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:50:10,941 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:50:10,941 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:50:10,942 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:50:10,943 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 17:50:10,973 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:50:11,219 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:11,492 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:50:11,492 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:50:11,492 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:50:11,492 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:50:11,492 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:50:11,493 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 17:50:11,519 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:11,520 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:50:11,632 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:50:11,635 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2523 [2023-09-07 17:50:11,638 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:50:11,638 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:50:11,638 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:50:11,639 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:50:11,639 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:50:11,645 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:50:11,647 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:50:11.647419 [2023-09-07 17:50:11,738 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:11,895 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:11,933 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:50:11,975 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:50:11,978 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:50:11,992 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:50:11,992 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109011.9318328, 'te': 1694109011.9419966, 'initial_status': True} ====================================================================================================  [2023-09-07 17:50:11,995 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:50:11.995259 [2023-09-07 17:50:12,003 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:50:12,004 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:50:12,077 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:12,186 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:50:12,187 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2544 [2023-09-07 17:50:12,187 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:50:12,187 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:50:12,188 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:50:12,188 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:50:12,188 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:50:12,194 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:50:12,196 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:50:12.196465 [2023-09-07 17:50:12,367 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:12,394 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:50:12,424 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:50:12,426 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:50:12,434 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:50:12,435 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109012.392353, 'te': 1694109012.4015267, 'initial_status': True} ====================================================================================================  [2023-09-07 17:50:12,436 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:50:12.436599 [2023-09-07 17:50:12,441 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:50:12,442 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:50:12,853 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:50:12,853 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:50:12,856 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:50:12.856624 [2023-09-07 17:50:12,864 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:50:12,864 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:50:12,871 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:50:12,877 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109012.8627014, 'te': 1694109012.8767858, 'initial_status': True} ====================================================================================================  [2023-09-07 17:50:12,880 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:50:12.880469 [2023-09-07 17:50:12,886 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:50:12,886 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:50:18,117 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 17:50:18,290 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:50:18,291 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:50:18,291 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:50:18,291 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:50:18,476 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:50:18,477 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:50:18,479 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 17:50:18,483 ERROR][11][wan-tasty-kitalpha][T_1] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:50:18,483 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:50:19,057 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109012, schain_finish_ts: 1694108304 [2023-09-07 17:50:19,480 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:50:20,204 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:50:20,204 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:50:20,204 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 17:50:21,189 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109012, schain_finish_ts: 1693742004 [2023-09-07 17:50:22,319 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:50:22,320 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:50:22,320 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:50:22,321 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 17:50:22,321 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 17:50:22,322 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:50:22,323 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:50:22,324 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:50:22,374 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:50:22,667 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:22,879 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:23,013 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:50:23,014 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2537 [2023-09-07 17:50:23,015 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:50:23,016 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:50:23,016 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:50:23,017 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:50:23,018 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:50:23,026 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:50:23,031 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:50:23.031499 [2023-09-07 17:50:23,259 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:23,284 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:50:23,314 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:50:23,316 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:50:23,324 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:50:23,324 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109023.2828035, 'te': 1694109023.2892935, 'initial_status': True} ====================================================================================================  [2023-09-07 17:50:23,326 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:50:23.326442 [2023-09-07 17:50:23,330 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:50:23,331 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:50:24,196 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:50:24,196 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:50:24,204 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:50:24.204429 [2023-09-07 17:50:24,210 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:50:32,863 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109012, schain_finish_ts: 1692377364 [2023-09-07 17:50:33,984 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:50:33,984 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:50:33,984 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:50:33,984 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:50:33,985 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:50:33,985 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:50:33,985 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:50:33,989 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 17:50:34,018 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:50:34,260 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:34,479 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:34,621 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:50:34,623 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2524 [2023-09-07 17:50:34,624 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:50:34,624 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:50:34,624 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:50:34,624 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:50:34,625 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:50:34,635 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:50:34,638 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:50:34.638577 [2023-09-07 17:50:34,809 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:34,834 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:50:34,876 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:50:34,878 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:50:34,887 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:50:34,888 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109034.8326, 'te': 1694109034.8406758, 'initial_status': True} ====================================================================================================  [2023-09-07 17:50:34,890 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:50:34.890534 [2023-09-07 17:50:34,896 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:50:34,897 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:50:35,427 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109024, schain_finish_ts: 1693742304 [2023-09-07 17:50:35,784 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:50:35,784 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:50:35,786 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:50:35.786493 [2023-09-07 17:50:35,794 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:50:35,794 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:50:35,800 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:50:35,805 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109035.7922642, 'te': 1694109035.805352, 'initial_status': True} ====================================================================================================  [2023-09-07 17:50:35,808 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:50:35.808158 [2023-09-07 17:50:35,813 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:50:35,813 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:50:36,552 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:50:36,552 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:50:36,554 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:50:36,555 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 17:50:36,556 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 17:50:36,556 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:50:36,562 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:50:36,562 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:50:36,596 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:50:36,751 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109024, schain_finish_ts: 1693412520 [2023-09-07 17:50:37,081 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:37,219 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:37,353 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:50:37,356 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2545 [2023-09-07 17:50:37,357 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:50:37,357 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:50:37,357 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:50:37,357 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:50:37,358 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:50:37,364 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:50:37,366 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:50:37.366157 [2023-09-07 17:50:37,555 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:37,594 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:50:37,628 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:50:37,631 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:50:37,640 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:50:37,641 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109037.5910504, 'te': 1694109037.598793, 'initial_status': True} ====================================================================================================  [2023-09-07 17:50:37,642 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:50:37.642660 [2023-09-07 17:50:37,648 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:50:37,649 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:50:37,882 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:50:37,882 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:50:37,883 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:50:37,884 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:50:37,884 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:50:37,885 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:50:37,888 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:50:37,888 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 17:50:37,916 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:50:38,227 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:38,515 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:50:38,516 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:50:38,518 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:50:38.518516 [2023-09-07 17:50:38,523 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:50:38,538 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:38,685 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:50:38,687 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2541 [2023-09-07 17:50:38,688 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:50:38,688 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:50:38,689 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:50:38,689 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:50:38,690 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:50:38,698 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:50:38,700 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:50:38.700318 [2023-09-07 17:50:39,068 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:39,116 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:50:39,167 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:50:39,170 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:50:39,178 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:50:39,180 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109039.110503, 'te': 1694109039.1256118, 'initial_status': True} ====================================================================================================  [2023-09-07 17:50:39,182 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:50:39.182381 [2023-09-07 17:50:39,188 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:50:39,191 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:50:39,676 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109036, schain_finish_ts: 0 [2023-09-07 17:50:39,755 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:50:39,756 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:50:39,759 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:50:39.759931 [2023-09-07 17:50:39,767 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:50:39,768 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:50:39,774 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:50:39,778 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109039.765963, 'te': 1694109039.7786052, 'initial_status': True} ====================================================================================================  [2023-09-07 17:50:39,780 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:50:39.780636 [2023-09-07 17:50:39,785 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:50:39,785 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:50:40,812 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:50:40,812 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:50:40,813 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:50:43,340 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 17:50:43,340 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 17:50:48,084 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109036, schain_finish_ts: 1692637668 [2023-09-07 17:50:49,263 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:50:49,263 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:50:49,264 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:50:49,264 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:50:49,264 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:50:49,264 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:50:49,266 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:50:49,268 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:50:49,297 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:50:49,596 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:49,946 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:50,103 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:50:50,104 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2540 [2023-09-07 17:50:50,104 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:50:50,105 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:50:50,105 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:50:50,105 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:50:50,105 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:50:50,111 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:50:50,113 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:50:50.113716 [2023-09-07 17:50:50,346 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:50,378 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:50:50,378 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:50:50,408 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:50:50,410 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:50:50,417 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:50:50,417 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109050.375941, 'te': 1694109050.3850014, 'initial_status': True} ====================================================================================================  [2023-09-07 17:50:50,418 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:50:50.418972 [2023-09-07 17:50:50,423 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:50:50,424 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:50:51,188 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:50:51,189 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:50:51,191 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:50:51.191109 [2023-09-07 17:50:51,197 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:50:51,197 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:50:51,204 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:50:51,209 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109051.1960225, 'te': 1694109051.2090175, 'initial_status': True} ====================================================================================================  [2023-09-07 17:50:51,211 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:50:51.211401 [2023-09-07 17:50:51,215 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:50:51,216 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:50:51,275 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109048, schain_finish_ts: 1693742004 [2023-09-07 17:50:52,416 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:50:52,417 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:50:52,417 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:50:52,417 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:50:52,417 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:50:52,417 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 17:50:52,439 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:50:52,782 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:52,947 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:53,077 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:50:53,078 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2538 [2023-09-07 17:50:53,079 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:50:53,079 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:50:53,079 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:50:53,079 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:50:53,080 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:50:53,086 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:50:53,089 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:50:53.089205 [2023-09-07 17:50:53,277 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:50:53,309 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:50:53,338 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:50:53,340 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:50:53,348 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:50:53,349 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109053.3064966, 'te': 1694109053.31451, 'initial_status': True} ====================================================================================================  [2023-09-07 17:50:53,350 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:50:53.350829 [2023-09-07 17:50:53,355 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:50:53,355 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:50:57,661 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 17:50:57,662 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 17:51:01,150 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109048, schain_finish_ts: 1694108304 [2023-09-07 17:51:02,294 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:51:02,294 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:51:02,295 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 17:51:03,239 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 17:51:03,423 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:51:03,423 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:51:03,424 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:51:03,424 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:51:03,605 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:51:03,606 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:51:03,608 INFO][8][curly-red-alterf][T_0] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 17:51:03,615 ERROR][8][curly-red-alterf][T_0] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:51:03,615 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:51:09,869 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109060, schain_finish_ts: 1693412520 [2023-09-07 17:51:11,002 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:51:11,002 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:51:11,003 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:51:11,003 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:51:11,003 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:51:11,003 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:51:11,005 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:51:11,005 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 17:51:11,036 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:51:11,406 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:11,575 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:11,698 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:51:11,700 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2542 [2023-09-07 17:51:11,701 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:51:11,701 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:51:11,701 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:51:11,701 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:51:11,702 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:51:11,708 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:51:11,711 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:51:11.710999 [2023-09-07 17:51:11,903 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:11,934 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:51:11,963 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:51:11,965 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:51:11,972 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:51:11,972 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109071.9328365, 'te': 1694109071.9397478, 'initial_status': True} ====================================================================================================  [2023-09-07 17:51:11,974 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:51:11.974013 [2023-09-07 17:51:11,979 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:51:11,980 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:51:12,488 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109060, schain_finish_ts: 1693742304 [2023-09-07 17:51:12,577 INFO][1][MainProcess][MainThread] - core.schains.cleaner:65 - Cleaner process started [2023-09-07 17:51:12,614 INFO][30872][cleaner][MainThread] - core.schains.cleaner:110 - Cleaner procedure started. [2023-09-07 17:51:12,835 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:51:12,835 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:51:12,837 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:51:12.837930 [2023-09-07 17:51:12,845 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:51:12,845 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:51:12,851 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:51:12,856 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109072.844176, 'te': 1694109072.8559089, 'initial_status': True} ====================================================================================================  [2023-09-07 17:51:12,857 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:51:12.857856 [2023-09-07 17:51:12,863 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:51:12,864 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:51:13,610 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:51:13,610 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:51:13,610 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:51:13,611 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:51:13,611 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:51:13,611 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:51:13,638 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:51:13,877 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:14,093 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:14,206 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:51:14,207 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2546 [2023-09-07 17:51:14,208 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:51:14,208 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:51:14,208 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:51:14,208 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:51:14,209 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:51:14,215 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:51:14,217 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:51:14.217303 [2023-09-07 17:51:14,407 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:14,433 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:51:14,466 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:51:14,468 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:51:14,477 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:51:14,477 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109074.4320588, 'te': 1694109074.4383929, 'initial_status': True} ====================================================================================================  [2023-09-07 17:51:14,479 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:51:14.479005 [2023-09-07 17:51:14,483 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:51:14,483 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:51:14,950 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109072, schain_finish_ts: 1692377364 [2023-09-07 17:51:16,106 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:51:16,106 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:51:16,106 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:51:16,107 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:51:16,107 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:51:16,107 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:51:16,109 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:51:16,111 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 17:51:16,161 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:51:16,590 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:16,981 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:17,095 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 17:51:17,100 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:51:17,101 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2525 [2023-09-07 17:51:17,102 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:51:17,102 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:51:17,102 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:51:17,103 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:51:17,103 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:51:17,112 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:51:17,129 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:51:17.129862 [2023-09-07 17:51:17,286 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:51:17,287 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:51:17,287 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:51:17,287 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:51:17,300 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:17,332 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:51:17,373 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:51:17,376 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:51:17,389 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:51:17,391 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109077.328513, 'te': 1694109077.3387027, 'initial_status': True} ====================================================================================================  [2023-09-07 17:51:17,396 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:51:17.396368 [2023-09-07 17:51:17,403 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:51:17,404 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:51:17,452 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:51:17,453 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:51:17,455 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 17:51:17,460 ERROR][11][wan-tasty-kitalpha][T_0] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:51:17,461 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:51:17,951 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:51:17,952 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:51:17,958 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:51:17.958553 [2023-09-07 17:51:17,975 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:51:17,976 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:51:17,984 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:51:17,988 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109077.9684067, 'te': 1694109077.988696, 'initial_status': True} ====================================================================================================  [2023-09-07 17:51:17,990 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:51:17.990817 [2023-09-07 17:51:17,996 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:51:17,996 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:51:18,277 INFO][30872][cleaner][MainThread] - core.schains.cleaner:116 - sChains on contracts: ['portly-passionate-sirius', 'curly-red-alterf', 'howling-hoedus', 'green-giddy-denebola', 'wan-tasty-kitalpha', 'free-basic-rosy-SKALE', 'submissive-well-off-rotanev'] sChains on node: ['curly-red-alterf', 'free-basic-rosy-SKALE', 'green-giddy-denebola', 'howling-hoedus', 'portly-passionate-sirius', 'submissive-well-off-rotanev', 'wan-tasty-kitalpha'] [2023-09-07 17:51:18,278 INFO][30872][cleaner][MainThread] - core.schains.cleaner:132 - Cleanup procedure finished [2023-09-07 17:51:18,282 INFO][1][MainProcess][MainThread] - core.schains.cleaner:67 - Cleaner process is joined. [2023-09-07 17:51:18,283 INFO][1][MainProcess][MainThread] - core.schains.cleaner:68 - Terminating the process [2023-09-07 17:51:18,284 INFO][1][MainProcess][MainThread] - __main__:70 - Sleeping for 90s after run_cleaner [2023-09-07 17:51:20,794 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109072, schain_finish_ts: 0 [2023-09-07 17:51:21,322 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:51:21,985 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:51:21,985 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:51:21,985 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 17:51:23,273 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:40 - Collecting rotation history for submissive-well-off-rotanev... [2023-09-07 17:51:24,443 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:50 - Rotation data for submissive-well-off-rotanev: Rotation(leaving_node_id=65, new_node_id=65, freeze_until=1694108100, rotation_counter=8) [2023-09-07 17:51:28,249 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109084, schain_finish_ts: 1692637668 [2023-09-07 17:51:29,260 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109084, schain_finish_ts: 1694108304 [2023-09-07 17:51:29,404 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:51:29,404 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:51:29,405 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:51:29,405 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:51:29,405 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:51:29,406 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:51:29,406 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:51:29,409 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 17:51:29,432 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:51:29,638 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:29,801 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:29,908 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:51:29,909 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2541 [2023-09-07 17:51:29,909 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:51:29,909 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:51:29,911 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:51:29,911 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:51:29,911 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:51:29,918 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:51:29,920 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:51:29.920321 [2023-09-07 17:51:30,117 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:30,149 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:51:30,199 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:51:30,201 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:51:30,209 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:51:30,209 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109090.147955, 'te': 1694109090.1559713, 'initial_status': True} ====================================================================================================  [2023-09-07 17:51:30,211 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:51:30.211600 [2023-09-07 17:51:30,215 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:51:30,216 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:51:30,404 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:51:30,405 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:51:30,405 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:51:31,178 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:51:31,179 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:51:31,181 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:51:31.181005 [2023-09-07 17:51:31,188 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:51:31,189 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:51:31,195 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:51:31,199 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109091.1870632, 'te': 1694109091.199518, 'initial_status': True} ====================================================================================================  [2023-09-07 17:51:31,201 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:51:31.201379 [2023-09-07 17:51:31,206 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:51:31,207 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:51:33,362 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109084, schain_finish_ts: 1693742004 [2023-09-07 17:51:34,497 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:51:34,498 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:51:34,498 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:51:34,499 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 17:51:34,499 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 17:51:34,500 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:51:34,505 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:51:34,508 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 17:51:34,548 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:51:34,795 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:34,933 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:35,054 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:51:35,063 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2539 [2023-09-07 17:51:35,064 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:51:35,065 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:51:35,065 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:51:35,066 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:51:35,070 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:51:35,076 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:51:35,078 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:51:35.078768 [2023-09-07 17:51:35,292 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:35,330 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:51:35,377 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:51:35,380 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:51:35,403 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:51:35,404 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109095.328844, 'te': 1694109095.3401723, 'initial_status': True} ====================================================================================================  [2023-09-07 17:51:35,408 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:51:35.408660 [2023-09-07 17:51:35,414 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:51:35,415 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:51:36,253 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:51:36,253 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:51:36,255 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:51:36.255616 [2023-09-07 17:51:36,261 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:51:39,639 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 63 -> 65 [2023-09-07 17:51:40,090 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109084, schain_finish_ts: 1692377364 [2023-09-07 17:51:41,220 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:51:41,220 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:51:41,221 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:51:41,221 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:51:41,221 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:51:41,222 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:51:41,224 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:51:41,224 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 17:51:41,255 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:51:41,518 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:41,702 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:41,824 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:51:41,825 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2526 [2023-09-07 17:51:41,826 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:51:41,826 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:51:41,827 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:51:41,827 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:51:41,827 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:51:41,852 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:51:41,854 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:51:41.854476 [2023-09-07 17:51:42,069 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:42,097 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:51:42,141 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:51:42,144 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:51:42,152 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:51:42,153 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109102.0959365, 'te': 1694109102.1036866, 'initial_status': True} ====================================================================================================  [2023-09-07 17:51:42,154 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:51:42.154610 [2023-09-07 17:51:42,159 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:51:42,159 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:51:43,057 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:51:43,058 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:51:43,060 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:51:43.060394 [2023-09-07 17:51:43,068 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:51:43,068 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:51:43,076 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:51:43,084 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109103.0670233, 'te': 1694109103.083881, 'initial_status': True} ====================================================================================================  [2023-09-07 17:51:43,087 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:51:43.087291 [2023-09-07 17:51:43,092 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:51:43,092 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:51:44,539 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 23 -> 24 [2023-09-07 17:51:44,957 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109096, schain_finish_ts: 1693412520 [2023-09-07 17:51:45,919 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109096, schain_finish_ts: 0 [2023-09-07 17:51:46,097 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:51:46,098 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:51:46,098 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:51:46,098 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:51:46,098 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:51:46,099 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:51:46,100 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:51:46,102 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:51:46,126 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:51:46,351 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:46,527 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:46,645 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:51:46,646 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2543 [2023-09-07 17:51:46,647 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:51:46,647 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:51:46,647 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:51:46,647 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:51:46,648 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:51:46,654 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:51:46,656 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:51:46.656107 [2023-09-07 17:51:46,988 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:47,026 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:51:47,047 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:51:47,048 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:51:47,048 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 17:51:47,064 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:51:47,066 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:51:47,074 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:51:47,074 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109107.0241601, 'te': 1694109107.032687, 'initial_status': True} ====================================================================================================  [2023-09-07 17:51:47,076 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:51:47.076736 [2023-09-07 17:51:47,081 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:51:47,083 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:51:47,950 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:51:47,950 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:51:47,953 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:51:47.953154 [2023-09-07 17:51:47,961 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:51:47,961 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:51:47,968 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:51:47,973 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109107.9594386, 'te': 1694109107.9729972, 'initial_status': True} ====================================================================================================  [2023-09-07 17:51:47,975 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:51:47.975244 [2023-09-07 17:51:47,979 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:51:47,980 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:51:49,170 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 9 -> 64 [2023-09-07 17:51:51,605 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109108, schain_finish_ts: 1693742304 [2023-09-07 17:51:52,280 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:51:52,754 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:51:52,754 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:51:52,755 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:51:52,755 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 17:51:52,755 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 17:51:52,755 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:51:52,757 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:51:52,759 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:51:52,795 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:51:53,176 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:53,369 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:53,503 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:51:53,504 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2547 [2023-09-07 17:51:53,504 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:51:53,505 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:51:53,505 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:51:53,505 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:51:53,505 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:51:53,514 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:51:53,517 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:51:53.517314 [2023-09-07 17:51:53,783 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:53,809 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:51:53,849 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:51:53,851 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:51:53,858 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:51:53,859 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109113.8073535, 'te': 1694109113.8173575, 'initial_status': True} ====================================================================================================  [2023-09-07 17:51:53,860 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:51:53.860941 [2023-09-07 17:51:53,866 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:51:53,867 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:51:54,194 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 35 -> 16 [2023-09-07 17:51:54,619 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:51:54,619 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:51:54,621 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:51:54.621511 [2023-09-07 17:51:54,626 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:51:55,050 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 17:51:55,051 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 17:51:57,358 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109108, schain_finish_ts: 1692637668 [2023-09-07 17:51:58,477 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:51:58,477 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:51:58,478 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:51:58,478 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:51:58,478 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:51:58,478 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:51:58,480 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:51:58,482 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:51:58,510 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:51:58,728 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:58,903 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:59,057 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:51:59,058 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2542 [2023-09-07 17:51:59,058 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:51:59,059 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:51:59,059 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:51:59,059 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:51:59,059 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:51:59,065 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:51:59,070 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:51:59.070604 [2023-09-07 17:51:59,331 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:51:59,361 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:51:59,393 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:51:59,395 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:51:59,404 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:51:59,404 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109119.3566992, 'te': 1694109119.3660982, 'initial_status': True} ====================================================================================================  [2023-09-07 17:51:59,406 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:51:59.406014 [2023-09-07 17:51:59,414 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:51:59,415 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:51:59,531 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 55 -> 63 [2023-09-07 17:52:00,271 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:52:00,272 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:52:00,275 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:52:00.275646 [2023-09-07 17:52:00,284 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:52:00,284 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:52:00,291 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:52:00,297 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109120.28232, 'te': 1694109120.2968485, 'initial_status': True} ====================================================================================================  [2023-09-07 17:52:00,300 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:52:00.300338 [2023-09-07 17:52:00,306 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:52:00,307 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:52:02,445 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109120, schain_finish_ts: 1693742004 [2023-09-07 17:52:03,575 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:52:03,575 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:52:03,575 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:52:03,575 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:52:03,576 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:52:03,576 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:52:03,609 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:52:03,886 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:04,083 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:04,207 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:52:04,208 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2540 [2023-09-07 17:52:04,209 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:52:04,209 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:52:04,210 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:52:04,210 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:52:04,210 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:52:04,216 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:52:04,218 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:52:04.218763 [2023-09-07 17:52:04,387 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:04,421 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:52:04,454 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:52:04,456 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:52:04,464 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:52:04,464 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109124.4202032, 'te': 1694109124.427709, 'initial_status': True} ====================================================================================================  [2023-09-07 17:52:04,466 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:52:04.466040 [2023-09-07 17:52:04,470 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:52:04,471 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:52:05,354 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 14 -> 11 [2023-09-07 17:52:06,343 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109120, schain_finish_ts: 1694108304 [2023-09-07 17:52:07,457 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:52:07,458 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:52:07,458 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:52:08,974 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109120, schain_finish_ts: 0 [2023-09-07 17:52:10,119 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:52:10,119 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:52:10,120 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 17:52:11,413 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 56 -> 60 [2023-09-07 17:52:13,383 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 17:52:13,387 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 17:52:14,720 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 17:52:14,904 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:52:14,904 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:52:14,905 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:52:14,905 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:52:15,039 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109120, schain_finish_ts: 1693412520 [2023-09-07 17:52:15,078 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:52:15,079 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:52:15,084 INFO][8][curly-red-alterf][T_1] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 17:52:15,089 ERROR][8][curly-red-alterf][T_1] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:52:15,090 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:52:15,164 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109120, schain_finish_ts: 1692377364 [2023-09-07 17:52:16,180 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:52:16,180 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:52:16,180 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:52:16,181 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:52:16,181 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:52:16,182 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:52:16,184 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:52:16,190 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 17:52:16,220 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:52:16,296 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:52:16,297 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:52:16,298 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:52:16,299 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:52:16,299 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:52:16,300 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:52:16,304 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:52:16,307 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 17:52:16,337 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:52:16,544 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:16,734 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:16,914 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:17,078 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:52:17,080 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2544 [2023-09-07 17:52:17,081 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:52:17,081 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:52:17,082 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:52:17,082 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:52:17,082 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:52:17,090 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:52:17,093 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:52:17.093598 [2023-09-07 17:52:17,235 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:17,377 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:52:17,378 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2527 [2023-09-07 17:52:17,379 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:52:17,383 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:52:17,384 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:52:17,385 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:52:17,386 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:52:17,398 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:52:17,403 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:52:17.403426 [2023-09-07 17:52:17,423 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:17,455 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:52:17,490 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:52:17,493 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:52:17,502 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:52:17,503 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109137.4531589, 'te': 1694109137.4604864, 'initial_status': True} ====================================================================================================  [2023-09-07 17:52:17,505 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:52:17.505184 [2023-09-07 17:52:17,510 ERROR][24912][submissive-well-off-rotanev][T_0] - core.schains.task:26 - Task submissive-well-off-rotanev-config failed with max() arg is an empty sequence Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 63, in execute self.am.upstream_config() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 194, in upstream_config new_config = create_new_upstream_config( File "/usr/src/admin/core/schains/config/main.py", line 51, in create_new_upstream_config schain_config = generate_schain_config_with_skale( File "/usr/src/admin/core/schains/config/generator.py", line 213, in generate_schain_config_with_skale node_groups = get_previous_schain_groups(skale, schain_name) File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 56, in get_previous_schain_groups _add_previous_schain_rotations_state( File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 119, in _add_previous_schain_rotations_state new_node_id = max(previous_nodes.items(), key=lambda x: x[1]['finish_ts'])[0] ValueError: max() arg is an empty sequence [2023-09-07 17:52:17,511 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:52:17,512 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:52:17,517 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:52:17,565 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:17,597 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:52:17,630 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:52:17,632 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:52:17,640 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:52:17,641 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109137.5952566, 'te': 1694109137.6033819, 'initial_status': True} ====================================================================================================  [2023-09-07 17:52:17,643 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:52:17.643825 [2023-09-07 17:52:17,648 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:52:17,649 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:52:17,947 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:52:17,948 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:52:17,950 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:52:17.950878 [2023-09-07 17:52:17,958 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:52:17,959 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:52:17,965 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:52:17,972 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109137.9566543, 'te': 1694109137.9720736, 'initial_status': True} ====================================================================================================  [2023-09-07 17:52:17,975 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:52:17.975314 [2023-09-07 17:52:17,979 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:52:17,980 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:52:18,132 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:52:18,132 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:52:18,134 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:52:18.134728 [2023-09-07 17:52:18,141 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:52:18,141 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:52:18,147 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:52:18,152 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109138.1400678, 'te': 1694109138.1520116, 'initial_status': True} ====================================================================================================  [2023-09-07 17:52:18,154 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:52:18.154650 [2023-09-07 17:52:18,158 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:52:18,159 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:52:21,416 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109120, schain_finish_ts: 1692637668 [2023-09-07 17:52:22,524 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:52:22,524 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:52:22,524 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:52:22,524 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:52:22,524 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:52:22,525 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:52:22,527 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:52:22,528 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:52:22,564 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:52:22,758 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:22,949 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:23,065 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:52:23,066 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2543 [2023-09-07 17:52:23,067 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:52:23,067 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:52:23,067 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:52:23,068 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:52:23,068 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:52:23,074 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:52:23,076 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:52:23.076147 [2023-09-07 17:52:23,229 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:23,245 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:52:23,256 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:52:23,301 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:52:23,303 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:52:23,310 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:52:23,311 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109143.2551363, 'te': 1694109143.2612917, 'initial_status': True} ====================================================================================================  [2023-09-07 17:52:23,312 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:52:23.312509 [2023-09-07 17:52:23,317 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:52:23,317 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:52:24,362 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:52:24,362 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:52:24,364 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:52:24.364781 [2023-09-07 17:52:24,371 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:52:24,372 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:52:24,378 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:52:24,383 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109144.3703744, 'te': 1694109144.3834088, 'initial_status': True} ====================================================================================================  [2023-09-07 17:52:24,385 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:52:24.385364 [2023-09-07 17:52:24,389 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:52:24,390 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:52:28,742 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109144, schain_finish_ts: 1693742304 [2023-09-07 17:52:29,892 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:52:29,892 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:52:29,892 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:52:29,893 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:52:29,893 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:52:29,893 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:52:29,916 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:52:30,258 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:30,480 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:30,639 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:52:30,646 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2548 [2023-09-07 17:52:30,661 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:52:30,661 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:52:30,661 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:52:30,662 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:52:30,663 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:52:30,680 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:52:30,688 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:52:30.688391 [2023-09-07 17:52:31,167 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:31,203 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:52:31,251 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:52:31,253 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:52:31,270 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:52:31,271 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109151.199199, 'te': 1694109151.2084985, 'initial_status': True} ====================================================================================================  [2023-09-07 17:52:31,275 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:52:31.275192 [2023-09-07 17:52:31,280 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:52:31,281 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:52:32,794 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 17:52:32,957 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:52:32,957 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:52:32,957 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:52:32,958 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:52:33,133 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:52:33,133 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:52:33,135 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 17:52:33,148 ERROR][11][wan-tasty-kitalpha][T_1] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:52:33,149 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:52:34,101 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109144, schain_finish_ts: 0 [2023-09-07 17:52:35,228 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:52:35,229 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:52:35,229 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:40 - Task submissive-well-off-rotanev-config finished with None [2023-09-07 17:52:35,229 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:42 - Running task submissive-well-off-rotanev-config [2023-09-07 17:52:35,230 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 17:52:36,986 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': False} [2023-09-07 17:52:36,986 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:52:36,988 INFO][24912][submissive-well-off-rotanev][T_0] - web.models.schain:152 - Changing monitor_last_seen for submissive-well-off-rotanev to 2023-09-07 17:52:36.988308 [2023-09-07 17:52:36,993 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.action:190 - Creating new upstream_config rotation_id: 8, stream: regression-2.2 [2023-09-07 17:52:36,995 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.config.main:49 - Generating sChain config for submissive-well-off-rotanev [2023-09-07 17:52:41,554 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109144, schain_finish_ts: 1693742004 [2023-09-07 17:52:42,702 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:52:42,703 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:52:42,703 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:52:42,703 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 17:52:42,703 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 17:52:42,704 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:52:42,704 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:52:42,704 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 17:52:42,735 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:52:42,943 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:43,099 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:43,215 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:52:43,216 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2541 [2023-09-07 17:52:43,217 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:52:43,217 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:52:43,217 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:52:43,217 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:52:43,218 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:52:43,224 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:52:43,226 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:52:43.226417 [2023-09-07 17:52:43,399 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109144, schain_finish_ts: 1694108304 [2023-09-07 17:52:43,522 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:43,547 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:52:43,584 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:52:43,586 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:52:43,593 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:52:43,593 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109163.546116, 'te': 1694109163.5529761, 'initial_status': True} ====================================================================================================  [2023-09-07 17:52:43,594 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:52:43.594911 [2023-09-07 17:52:43,599 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:52:43,600 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:52:44,478 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:52:44,479 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:52:44,482 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:52:44.482400 [2023-09-07 17:52:44,491 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:52:44,528 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:52:44,529 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:52:44,529 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 17:52:46,121 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109144, schain_finish_ts: 1693412520 [2023-09-07 17:52:47,258 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:52:47,259 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:52:47,259 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:52:47,259 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:52:47,259 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:52:47,259 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:52:47,260 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:52:47,260 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 17:52:47,286 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:52:47,594 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:47,747 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:47,868 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:52:47,869 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2545 [2023-09-07 17:52:47,870 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:52:47,870 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:52:47,871 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:52:47,871 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:52:47,871 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:52:47,878 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:52:47,881 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:52:47.880992 [2023-09-07 17:52:48,135 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:48,164 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:52:48,203 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:52:48,204 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:52:48,215 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:52:48,216 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109168.1623642, 'te': 1694109168.171785, 'initial_status': True} ====================================================================================================  [2023-09-07 17:52:48,218 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:52:48.218579 [2023-09-07 17:52:48,224 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:52:48,225 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:52:48,374 INFO][1][MainProcess][MainThread] - core.schains.process_manager:57 - Process manager started [2023-09-07 17:52:48,604 INFO][1][MainProcess][MainThread] - core.schains.notifications:35 - Node account has 406903463189434189 WEI [2023-09-07 17:52:48,608 INFO][1][MainProcess][MainThread] - tools.notifications.messages:188 - Saving new balance state 3016 1 [2023-09-07 17:52:48,611 INFO][1][MainProcess][MainThread] - core.schains.process_manager:95 - Fetching schains to monitor... [2023-09-07 17:52:49,048 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:52:49,049 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:52:49,050 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:52:49.050827 [2023-09-07 17:52:49,058 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:52:49,058 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:52:49,063 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:52:49,068 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109169.0568876, 'te': 1694109169.0682452, 'initial_status': True} ====================================================================================================  [2023-09-07 17:52:49,070 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:52:49.070000 [2023-09-07 17:52:49,074 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:52:49,074 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:52:51,252 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109168, schain_finish_ts: 1692377364 [2023-09-07 17:52:51,488 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109168, schain_finish_ts: 1692637668 [2023-09-07 17:52:52,394 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:52:52,394 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:52:52,394 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:52:52,394 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:52:52,395 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:52:52,395 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:52:52,397 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:52:52,397 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 17:52:52,464 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:52:52,636 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:52:52,636 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:52:52,637 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:52:52,637 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:52:52,638 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:52:52,638 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:52:52,641 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:52:52,641 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:52:52,672 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:52:52,696 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:52,859 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:52,984 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:52:52,985 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2528 [2023-09-07 17:52:52,986 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:52:52,987 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:52:52,988 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:52:52,989 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:52:52,989 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:52:53,004 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:52:53,007 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:52:53.007443 [2023-09-07 17:52:53,074 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:53,247 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:53,274 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:52:53,308 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:52:53,310 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:52:53,321 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:52:53,322 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109173.2722483, 'te': 1694109173.280879, 'initial_status': True} ====================================================================================================  [2023-09-07 17:52:53,325 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:52:53.325426 [2023-09-07 17:52:53,332 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:52:53,333 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:52:53,596 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:53,720 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:52:53,721 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2544 [2023-09-07 17:52:53,722 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:52:53,722 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:52:53,722 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:52:53,723 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:52:53,723 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:52:53,729 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:52:53,730 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:52:53.730963 [2023-09-07 17:52:53,922 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:52:53,948 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:52:53,981 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:52:53,983 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:52:53,993 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:52:53,993 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109173.9472318, 'te': 1694109173.954875, 'initial_status': True} ====================================================================================================  [2023-09-07 17:52:53,995 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:52:53.995309 [2023-09-07 17:52:53,999 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:52:54,000 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:52:54,216 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:52:54,221 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:52:54,221 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:52:54,223 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:52:54.223703 [2023-09-07 17:52:54,230 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:52:54,230 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:52:54,231 INFO][1][MainProcess][MainThread] - core.schains.process_manager:109 - Get leaving_history for node ... [2023-09-07 17:52:54,238 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:52:54,242 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109174.229065, 'te': 1694109174.2423246, 'initial_status': True} ====================================================================================================  [2023-09-07 17:52:54,244 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:52:54.244458 [2023-09-07 17:52:54,249 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:52:54,250 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:52:54,541 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:52:54,541 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:52:54,543 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:52:54.543548 [2023-09-07 17:52:54,551 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:52:54,551 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:52:54,558 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:52:54,564 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109174.549586, 'te': 1694109174.564282, 'initial_status': True} ====================================================================================================  [2023-09-07 17:52:54,566 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:52:54.566854 [2023-09-07 17:52:54,571 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:52:54,571 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:52:56,227 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109168, schain_finish_ts: 1692377616 [2023-09-07 17:52:58,221 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109168, schain_finish_ts: 0 [2023-09-07 17:52:58,595 INFO][1][MainProcess][MainThread] - core.schains.process_manager:117 - Got leaving sChains for the node: [] [2023-09-07 17:52:58,595 INFO][1][MainProcess][MainThread] - core.schains.process_manager:101 -  ==================================================================================================== Monitoring sChains Node ID: 16 sChains on node: [{'name': 'portly-passionate-sirius', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666281384, 'startBlock': 7803380, 'deposit': 0, 'index': 1, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x0000017723ca2', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'curly-red-alterf', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1676058936, 'startBlock': 8469646, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000001fe6cdfa', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'howling-hoedus', 'mainnetOwner': '0x1026467C91fdA40aCc7e6935d1E52f67274FD669', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666181400, 'startBlock': 7796660, 'deposit': 0, 'index': 0, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0xe5a600f4c2608', 'multitransactionMode': False, 'thresholdEncryption': False, 'active': True}, {'name': 'green-giddy-denebola', 'mainnetOwner': '0x8e835d5d9E28ffD259E8Fa8988Cacfc333A582B4', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674565164, 'startBlock': 8368049, 'deposit': 0, 'index': 4, 'generation': 1, 'originator': '0xd2001DAb6898127Be2F167B548691C87251D13C3', 'chainId': '0x00000585eb4b1', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'wan-tasty-kitalpha', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674568392, 'startBlock': 8368271, 'deposit': 0, 'index': 5, 'generation': 1, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x00000152e7be9', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'free-basic-rosy-SKALE', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1693996920, 'startBlock': 9646443, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000002bb90439', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'submissive-well-off-rotanev', 'mainnetOwner': '0x487360568fE5Ee4AE5c75782D5F63e9935B0E512', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1692361104, 'startBlock': 9539053, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000003791fa10', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}] Number of sChains on node: 7 Empty sChain structs: 0 ====================================================================================================  [2023-09-07 17:52:58,968 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:52:58,968 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain portly-passionate-sirius - Process is running: PID = 7 [2023-09-07 17:52:59,357 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:52:59,357 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain curly-red-alterf - Process is running: PID = 8 [2023-09-07 17:52:59,738 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:52:59,738 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain howling-hoedus - Process is running: PID = 9 [2023-09-07 17:53:00,113 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:53:00,113 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain green-giddy-denebola - Process is running: PID = 10 [2023-09-07 17:53:00,482 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:53:00,482 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain wan-tasty-kitalpha - Process is running: PID = 11 [2023-09-07 17:53:00,851 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:53:00,852 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain free-basic-rosy-SKALE - Process is running: PID = 30622 [2023-09-07 17:53:01,198 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109168, schain_finish_ts: 0 [2023-09-07 17:53:01,223 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:53:01,224 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain submissive-well-off-rotanev - Process is running: PID = 24912 [2023-09-07 17:53:01,224 INFO][1][MainProcess][MainThread] - core.schains.process_manager:88 - Process manager procedure finished [2023-09-07 17:53:01,224 INFO][1][MainProcess][MainThread] - __main__:65 - Sleeping for 90s after run_process_manager [2023-09-07 17:53:02,326 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:53:02,327 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:53:02,328 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 17:53:03,437 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 17:53:03,438 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 17:53:05,847 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109180, schain_finish_ts: 1693742304 [2023-09-07 17:53:06,991 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:53:06,991 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:53:06,991 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:53:06,992 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 17:53:06,992 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 17:53:06,992 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:53:06,994 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:53:06,994 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:53:07,035 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:53:07,324 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:07,531 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:07,681 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:53:07,682 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2549 [2023-09-07 17:53:07,683 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:53:07,683 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:53:07,684 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:53:07,684 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:53:07,684 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:53:07,691 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:53:07,695 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:53:07.695557 [2023-09-07 17:53:07,890 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:07,923 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:53:07,963 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:53:07,966 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:53:07,984 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:53:07,985 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109187.9195626, 'te': 1694109187.9280498, 'initial_status': True} ====================================================================================================  [2023-09-07 17:53:07,986 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:53:07.986827 [2023-09-07 17:53:07,991 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:53:07,992 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:53:08,785 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:53:08,785 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:53:08,788 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:53:08.788404 [2023-09-07 17:53:08,793 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:53:12,665 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109180, schain_finish_ts: 1693742004 [2023-09-07 17:53:13,798 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:53:13,798 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:53:13,799 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:53:13,799 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:53:13,799 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:53:13,799 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 17:53:13,835 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:53:14,149 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:14,359 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:14,507 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:53:14,508 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2542 [2023-09-07 17:53:14,509 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:53:14,509 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:53:14,509 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:53:14,510 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:53:14,510 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:53:14,520 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:53:14,522 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:53:14.522108 [2023-09-07 17:53:14,714 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:14,746 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:53:14,781 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:53:14,783 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:53:14,792 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:53:14,793 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109194.7450385, 'te': 1694109194.750959, 'initial_status': True} ====================================================================================================  [2023-09-07 17:53:14,795 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:53:14.795209 [2023-09-07 17:53:14,799 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:53:14,800 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:53:15,231 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109192, schain_finish_ts: 1693412520 [2023-09-07 17:53:16,359 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:53:16,360 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:53:16,360 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:53:16,360 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:53:16,360 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:53:16,361 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:53:16,362 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:53:16,363 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 17:53:16,396 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:53:16,474 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109192, schain_finish_ts: 1694108304 [2023-09-07 17:53:16,759 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:17,015 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:17,140 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:53:17,141 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2546 [2023-09-07 17:53:17,142 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:53:17,142 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:53:17,142 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:53:17,143 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:53:17,143 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:53:17,149 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:53:17,152 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:53:17.151982 [2023-09-07 17:53:17,299 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:17,328 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:53:17,371 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:53:17,374 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:53:17,385 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:53:17,386 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109197.3263755, 'te': 1694109197.3344204, 'initial_status': True} ====================================================================================================  [2023-09-07 17:53:17,389 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:53:17.389520 [2023-09-07 17:53:17,397 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:53:17,399 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:53:17,613 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:53:17,613 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:53:17,614 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:53:18,151 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:53:18,151 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:53:18,154 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:53:18.154006 [2023-09-07 17:53:18,161 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:53:18,162 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:53:18,169 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:53:18,176 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109198.1597502, 'te': 1694109198.176475, 'initial_status': True} ====================================================================================================  [2023-09-07 17:53:18,178 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:53:18.178625 [2023-09-07 17:53:18,183 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:53:18,183 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:53:21,572 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109192, schain_finish_ts: 1692637668 [2023-09-07 17:53:22,699 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:53:22,700 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:53:22,700 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:53:22,700 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:53:22,700 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:53:22,701 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:53:22,714 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:53:22,715 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 17:53:22,750 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:53:22,971 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:22,977 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 17:53:23,142 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:53:23,143 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:53:23,143 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:53:23,143 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:53:23,175 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:23,305 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:53:23,305 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:53:23,308 INFO][8][curly-red-alterf][T_0] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 17:53:23,310 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:53:23,312 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2545 [2023-09-07 17:53:23,312 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:53:23,313 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:53:23,313 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:53:23,313 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:53:23,314 ERROR][8][curly-red-alterf][T_0] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:53:23,314 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:53:23,314 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:53:23,321 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:53:23,323 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:53:23.323837 [2023-09-07 17:53:23,504 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:23,544 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:53:23,578 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:53:23,581 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:53:23,588 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:53:23,589 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109203.5427356, 'te': 1694109203.5499396, 'initial_status': True} ====================================================================================================  [2023-09-07 17:53:23,591 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:53:23.591298 [2023-09-07 17:53:23,597 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:53:23,598 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:53:24,555 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:53:24,555 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:53:24,558 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:53:24.558823 [2023-09-07 17:53:24,569 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:53:24,570 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:53:24,578 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:53:24,583 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109204.5658233, 'te': 1694109204.5837057, 'initial_status': True} ====================================================================================================  [2023-09-07 17:53:24,586 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:53:24.586763 [2023-09-07 17:53:24,592 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:53:24,593 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:53:25,161 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:53:27,374 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109192, schain_finish_ts: 1692377364 [2023-09-07 17:53:27,902 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 17:53:27,902 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 17:53:28,522 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:53:28,522 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:53:28,523 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:53:28,523 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:53:28,523 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:53:28,523 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:53:28,525 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:53:28,525 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:53:28,554 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:53:28,937 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:29,161 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:29,284 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109192, schain_finish_ts: 0 [2023-09-07 17:53:29,343 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:53:29,344 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2529 [2023-09-07 17:53:29,348 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:53:29,348 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:53:29,348 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:53:29,349 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:53:29,349 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:53:29,367 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:53:29,369 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:53:29.369797 [2023-09-07 17:53:29,569 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:29,614 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:53:29,649 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:53:29,653 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:53:29,662 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:53:29,663 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109209.6120408, 'te': 1694109209.6197665, 'initial_status': True} ====================================================================================================  [2023-09-07 17:53:29,666 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:53:29.666261 [2023-09-07 17:53:29,671 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:53:29,673 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:53:30,408 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:53:30,409 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:53:30,411 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:53:30.411646 [2023-09-07 17:53:30,419 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:53:30,420 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:53:30,426 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:53:30,431 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109210.4183738, 'te': 1694109210.4310412, 'initial_status': True} ====================================================================================================  [2023-09-07 17:53:30,433 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:53:30.433148 [2023-09-07 17:53:30,436 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:53:30,437 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:53:30,437 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 17:53:30,437 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:53:30,438 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:53:35,975 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109192, schain_finish_ts: 1693742304 [2023-09-07 17:53:37,127 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:53:37,128 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:53:37,128 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:53:37,128 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:53:37,128 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:53:37,129 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:53:37,161 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:53:37,422 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:37,618 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:37,773 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:53:37,774 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2550 [2023-09-07 17:53:37,775 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:53:37,775 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:53:37,775 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:53:37,775 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:53:37,776 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:53:37,784 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:53:37,786 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:53:37.786824 [2023-09-07 17:53:38,061 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:38,089 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:53:38,129 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:53:38,135 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:53:38,153 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:53:38,154 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109218.0873182, 'te': 1694109218.0951235, 'initial_status': True} ====================================================================================================  [2023-09-07 17:53:38,156 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:53:38.156602 [2023-09-07 17:53:38,163 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:53:38,165 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:53:40,737 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109216, schain_finish_ts: 1693742004 [2023-09-07 17:53:41,892 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:53:41,892 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:53:41,892 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:53:41,893 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 17:53:41,893 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 17:53:41,893 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:53:41,893 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:53:41,895 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:53:41,921 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:53:42,111 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:42,431 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:42,541 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109216, schain_finish_ts: 1694108304 [2023-09-07 17:53:42,549 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:53:42,553 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2543 [2023-09-07 17:53:42,555 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:53:42,555 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:53:42,555 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:53:42,556 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:53:42,556 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:53:42,562 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:53:42,565 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:53:42.565022 [2023-09-07 17:53:42,729 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:42,760 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:53:42,790 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:53:42,792 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:53:42,799 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:53:42,799 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109222.759242, 'te': 1694109222.767785, 'initial_status': True} ====================================================================================================  [2023-09-07 17:53:42,801 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:53:42.801854 [2023-09-07 17:53:42,806 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:53:42,807 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:53:43,661 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:53:43,661 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:53:43,661 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 17:53:43,785 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:53:43,785 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:53:43,789 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:53:43.789280 [2023-09-07 17:53:43,794 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:53:47,311 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109216, schain_finish_ts: 1693412520 [2023-09-07 17:53:47,532 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 17:53:47,697 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:53:47,698 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:53:47,698 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:53:47,698 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:53:47,869 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:53:47,869 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:53:47,871 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 17:53:47,877 ERROR][11][wan-tasty-kitalpha][T_0] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:53:47,877 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:53:48,441 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:53:48,442 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:53:48,442 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:53:48,442 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:53:48,442 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:53:48,443 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:53:48,445 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:53:48,445 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:53:48,469 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:53:48,714 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:49,055 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:49,196 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:53:49,197 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2547 [2023-09-07 17:53:49,202 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:53:49,208 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:53:49,208 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:53:49,209 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:53:49,213 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:53:49,229 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:53:49,232 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:53:49.232111 [2023-09-07 17:53:49,414 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:49,443 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:53:49,473 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:53:49,476 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:53:49,496 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:53:49,498 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109229.441076, 'te': 1694109229.448687, 'initial_status': True} ====================================================================================================  [2023-09-07 17:53:49,500 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:53:49.500620 [2023-09-07 17:53:49,505 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:53:49,506 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:53:50,304 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:53:50,305 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:53:50,307 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:53:50.307076 [2023-09-07 17:53:50,315 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:53:50,316 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:53:50,322 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:53:50,327 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109230.3144662, 'te': 1694109230.3275573, 'initial_status': True} ====================================================================================================  [2023-09-07 17:53:50,329 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:53:50.329327 [2023-09-07 17:53:50,335 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:53:50,335 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:53:50,667 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109228, schain_finish_ts: 1692637668 [2023-09-07 17:53:51,795 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:53:51,795 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:53:51,795 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:53:51,795 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:53:51,795 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:53:51,796 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:53:51,797 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:53:51,801 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 17:53:51,832 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:53:52,181 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:52,343 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:52,450 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:53:52,451 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2546 [2023-09-07 17:53:52,452 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:53:52,452 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:53:52,452 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:53:52,453 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:53:52,453 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:53:52,458 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:53:52,460 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:53:52.460700 [2023-09-07 17:53:52,604 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:53:52,629 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:53:52,659 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:53:52,661 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:53:52,667 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:53:52,668 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109232.6274483, 'te': 1694109232.634512, 'initial_status': True} ====================================================================================================  [2023-09-07 17:53:52,669 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:53:52.669702 [2023-09-07 17:53:52,674 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:53:52,675 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:53:53,533 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:53:53,534 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:53:53,536 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:53:53.536299 [2023-09-07 17:53:53,544 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:53:53,544 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:53:53,550 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:53:53,555 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109233.5426304, 'te': 1694109233.5554295, 'initial_status': True} ====================================================================================================  [2023-09-07 17:53:53,557 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:53:53.557289 [2023-09-07 17:53:53,562 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:53:53,562 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:53:56,135 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:54:01,490 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109228, schain_finish_ts: 1692377364 [2023-09-07 17:54:02,629 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:54:02,629 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:54:02,629 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:54:02,630 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:54:02,630 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:54:02,630 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:54:02,632 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:54:02,633 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 17:54:02,658 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:54:02,829 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 17:54:02,829 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 17:54:02,889 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:03,049 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:03,219 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:54:03,221 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2530 [2023-09-07 17:54:03,223 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:54:03,224 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:54:03,224 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:54:03,225 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:54:03,226 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:54:03,237 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:54:03,241 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:54:03.241747 [2023-09-07 17:54:03,447 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:03,490 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:54:03,536 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:54:03,540 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:54:03,552 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:54:03,553 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109243.4850311, 'te': 1694109243.4979138, 'initial_status': True} ====================================================================================================  [2023-09-07 17:54:03,555 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:54:03.555352 [2023-09-07 17:54:03,560 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:54:03,561 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:54:04,386 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109240, schain_finish_ts: 0 [2023-09-07 17:54:04,388 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:54:04,389 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:54:04,391 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:54:04.391413 [2023-09-07 17:54:04,400 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:54:04,400 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:54:04,409 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:54:04,415 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109244.3982337, 'te': 1694109244.415401, 'initial_status': True} ====================================================================================================  [2023-09-07 17:54:04,417 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:54:04.417398 [2023-09-07 17:54:04,422 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:54:04,422 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:54:04,819 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109240, schain_finish_ts: 1693742004 [2023-09-07 17:54:05,570 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:54:05,570 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:54:05,571 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 17:54:05,607 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109240, schain_finish_ts: 1694108304 [2023-09-07 17:54:05,946 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:54:05,946 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:54:05,947 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:54:05,947 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:54:05,947 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:54:05,947 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 17:54:05,966 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:54:06,152 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:06,309 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:06,411 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:54:06,411 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2544 [2023-09-07 17:54:06,412 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:54:06,412 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:54:06,413 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:54:06,413 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:54:06,414 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:54:06,420 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:54:06,421 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:54:06.421922 [2023-09-07 17:54:06,582 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:06,606 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:54:06,642 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:54:06,644 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:54:06,654 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:54:06,655 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109246.6044097, 'te': 1694109246.6116805, 'initial_status': True} ====================================================================================================  [2023-09-07 17:54:06,656 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:54:06.656832 [2023-09-07 17:54:06,670 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:54:06,671 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:54:06,725 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:54:06,725 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:54:06,726 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 17:54:10,091 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109240, schain_finish_ts: 1693742304 [2023-09-07 17:54:11,261 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:54:11,261 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:54:11,261 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:54:11,262 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 17:54:11,262 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 17:54:11,262 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:54:11,267 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:54:11,268 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 17:54:11,295 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:54:11,371 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109240, schain_finish_ts: 1693412520 [2023-09-07 17:54:11,492 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:11,653 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:11,782 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:54:11,783 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2551 [2023-09-07 17:54:11,784 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:54:11,784 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:54:11,784 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:54:11,784 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:54:11,785 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:54:11,791 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:54:11,793 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:54:11.793562 [2023-09-07 17:54:11,974 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:12,011 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:54:12,048 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:54:12,050 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:54:12,069 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:54:12,069 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109252.0050805, 'te': 1694109252.0216281, 'initial_status': True} ====================================================================================================  [2023-09-07 17:54:12,071 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:54:12.071801 [2023-09-07 17:54:12,089 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:54:12,090 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:54:12,492 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:54:12,492 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:54:12,492 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:54:12,492 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:54:12,492 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:54:12,493 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:54:12,494 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:54:12,494 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 17:54:12,517 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:54:12,704 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:12,844 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:12,954 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:54:12,955 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2548 [2023-09-07 17:54:12,955 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:54:12,956 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:54:12,956 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:54:12,956 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:54:12,956 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:54:12,962 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:54:12,964 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:54:12.964302 [2023-09-07 17:54:13,021 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:54:13,021 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:54:13,024 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:54:13.024220 [2023-09-07 17:54:13,030 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:54:13,131 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:13,157 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:54:13,201 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:54:13,203 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:54:13,211 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:54:13,211 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109253.1557412, 'te': 1694109253.1626534, 'initial_status': True} ====================================================================================================  [2023-09-07 17:54:13,213 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:54:13.213006 [2023-09-07 17:54:13,218 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:54:13,218 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:54:14,292 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:54:14,293 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:54:14,297 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:54:14.297783 [2023-09-07 17:54:14,307 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:54:14,307 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:54:14,313 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:54:14,319 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109254.304879, 'te': 1694109254.3192916, 'initial_status': True} ====================================================================================================  [2023-09-07 17:54:14,321 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:54:14.321698 [2023-09-07 17:54:14,329 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:54:14,329 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:54:18,736 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109252, schain_finish_ts: 1692637668 [2023-09-07 17:54:19,873 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:54:19,874 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:54:19,874 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:54:19,874 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:54:19,874 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:54:19,875 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:54:19,876 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:54:19,876 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:54:19,903 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:54:20,094 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:20,252 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:20,356 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:54:20,357 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2547 [2023-09-07 17:54:20,357 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:54:20,358 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:54:20,358 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:54:20,358 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:54:20,358 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:54:20,364 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:54:20,366 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:54:20.366428 [2023-09-07 17:54:20,517 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:20,545 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:54:20,581 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:54:20,586 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:54:20,598 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:54:20,598 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109260.5441647, 'te': 1694109260.551093, 'initial_status': True} ====================================================================================================  [2023-09-07 17:54:20,600 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:54:20.600368 [2023-09-07 17:54:20,607 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:54:20,608 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:54:21,645 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:54:21,645 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:54:21,647 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:54:21.647860 [2023-09-07 17:54:21,655 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:54:21,655 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:54:21,662 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:54:21,670 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109261.654289, 'te': 1694109261.670346, 'initial_status': True} ====================================================================================================  [2023-09-07 17:54:21,674 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:54:21.674783 [2023-09-07 17:54:21,681 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:54:21,681 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:54:22,391 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 17:54:22,582 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:54:22,582 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:54:22,583 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:54:22,583 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:54:22,763 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:54:22,764 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:54:22,765 INFO][8][curly-red-alterf][T_1] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 17:54:22,771 ERROR][8][curly-red-alterf][T_1] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:54:22,772 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:54:27,106 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:54:27,521 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109264, schain_finish_ts: 0 [2023-09-07 17:54:28,688 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:54:28,688 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:54:28,689 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 17:54:28,823 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:40 - Collecting rotation history for submissive-well-off-rotanev... [2023-09-07 17:54:29,568 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109264, schain_finish_ts: 1692377364 [2023-09-07 17:54:29,978 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:50 - Rotation data for submissive-well-off-rotanev: Rotation(leaving_node_id=65, new_node_id=65, freeze_until=1694108100, rotation_counter=8) [2023-09-07 17:54:30,727 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:54:30,727 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:54:30,728 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:54:30,728 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:54:30,728 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:54:30,728 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:54:30,732 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:54:30,733 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 17:54:30,762 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:54:31,155 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:31,297 INFO][1][MainProcess][MainThread] - core.schains.cleaner:65 - Cleaner process started [2023-09-07 17:54:31,312 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:31,344 INFO][30891][cleaner][MainThread] - core.schains.cleaner:110 - Cleaner procedure started. [2023-09-07 17:54:31,455 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:54:31,456 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2531 [2023-09-07 17:54:31,456 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:54:31,457 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:54:31,457 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:54:31,457 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:54:31,458 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:54:31,464 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:54:31,465 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:54:31.465873 [2023-09-07 17:54:31,642 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:31,685 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:54:31,726 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:54:31,730 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:54:31,740 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:54:31,741 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109271.6840632, 'te': 1694109271.691479, 'initial_status': True} ====================================================================================================  [2023-09-07 17:54:31,743 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:54:31.743379 [2023-09-07 17:54:31,751 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:54:31,754 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:54:32,045 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 17:54:32,045 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 17:54:32,630 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:54:32,631 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:54:32,632 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:54:32.632921 [2023-09-07 17:54:32,640 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:54:32,640 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:54:32,647 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:54:32,652 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109272.639058, 'te': 1694109272.6519897, 'initial_status': True} ====================================================================================================  [2023-09-07 17:54:32,653 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:54:32.653734 [2023-09-07 17:54:32,658 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:54:32,658 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:54:33,419 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109264, schain_finish_ts: 1693412520 [2023-09-07 17:54:34,555 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:54:34,556 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:54:34,556 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:54:34,557 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:54:34,557 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:54:34,558 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:54:34,562 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:54:34,562 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 17:54:34,601 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:54:34,935 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:35,080 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:35,227 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:54:35,228 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2549 [2023-09-07 17:54:35,228 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:54:35,229 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:54:35,229 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:54:35,229 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:54:35,229 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:54:35,235 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:54:35,237 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:54:35.237285 [2023-09-07 17:54:35,411 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:35,441 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:54:35,476 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:54:35,479 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:54:35,488 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:54:35,488 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109275.4394834, 'te': 1694109275.4473631, 'initial_status': True} ====================================================================================================  [2023-09-07 17:54:35,490 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:54:35.490365 [2023-09-07 17:54:35,494 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:54:35,495 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:54:36,449 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:54:36,449 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:54:36,452 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:54:36.452016 [2023-09-07 17:54:36,460 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:54:36,461 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:54:36,467 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:54:36,472 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109276.4590724, 'te': 1694109276.4725645, 'initial_status': True} ====================================================================================================  [2023-09-07 17:54:36,475 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:54:36.475424 [2023-09-07 17:54:36,481 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:54:36,482 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:54:37,035 INFO][30891][cleaner][MainThread] - core.schains.cleaner:116 - sChains on contracts: ['portly-passionate-sirius', 'curly-red-alterf', 'howling-hoedus', 'green-giddy-denebola', 'wan-tasty-kitalpha', 'free-basic-rosy-SKALE', 'submissive-well-off-rotanev'] sChains on node: ['curly-red-alterf', 'free-basic-rosy-SKALE', 'green-giddy-denebola', 'howling-hoedus', 'portly-passionate-sirius', 'submissive-well-off-rotanev', 'wan-tasty-kitalpha'] [2023-09-07 17:54:37,036 INFO][30891][cleaner][MainThread] - core.schains.cleaner:132 - Cleanup procedure finished [2023-09-07 17:54:37,040 INFO][1][MainProcess][MainThread] - core.schains.cleaner:67 - Cleaner process is joined. [2023-09-07 17:54:37,040 INFO][1][MainProcess][MainThread] - core.schains.cleaner:68 - Terminating the process [2023-09-07 17:54:37,041 INFO][1][MainProcess][MainThread] - __main__:70 - Sleeping for 90s after run_cleaner [2023-09-07 17:54:39,914 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109276, schain_finish_ts: 1693742004 [2023-09-07 17:54:41,070 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:54:41,070 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:54:41,070 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:54:41,070 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 17:54:41,071 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 17:54:41,071 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:54:41,073 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:54:41,073 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 17:54:41,109 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:54:41,362 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:41,611 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:41,718 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:54:41,719 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2545 [2023-09-07 17:54:41,719 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:54:41,720 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:54:41,720 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:54:41,720 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:54:41,721 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:54:41,731 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:54:41,734 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:54:41.734526 [2023-09-07 17:54:41,927 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:41,953 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:54:41,994 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:54:41,999 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:54:42,008 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:54:42,008 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109281.9513092, 'te': 1694109281.9584439, 'initial_status': True} ====================================================================================================  [2023-09-07 17:54:42,010 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:54:42.010735 [2023-09-07 17:54:42,016 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:54:42,017 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:54:42,916 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:54:42,916 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:54:42,919 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:54:42.919664 [2023-09-07 17:54:42,925 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:54:43,693 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109276, schain_finish_ts: 1694108304 [2023-09-07 17:54:44,840 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:54:44,840 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:54:44,840 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 17:54:45,229 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109276, schain_finish_ts: 1693742304 [2023-09-07 17:54:45,448 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 63 -> 65 [2023-09-07 17:54:46,373 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:54:46,374 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:54:46,374 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:54:46,374 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:54:46,374 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:54:46,375 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 17:54:46,410 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:54:46,717 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:47,161 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:47,297 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:54:47,298 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2552 [2023-09-07 17:54:47,299 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:54:47,299 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:54:47,299 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:54:47,299 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:54:47,300 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:54:47,306 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:54:47,308 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:54:47.308279 [2023-09-07 17:54:47,514 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:47,548 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:54:47,592 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:54:47,594 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:54:47,602 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:54:47,602 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109287.5461967, 'te': 1694109287.5532093, 'initial_status': True} ====================================================================================================  [2023-09-07 17:54:47,604 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:54:47.604459 [2023-09-07 17:54:47,609 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:54:47,610 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:54:50,431 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 23 -> 24 [2023-09-07 17:54:51,654 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 17:54:51,817 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:54:51,817 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:54:51,817 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:54:51,818 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:54:52,005 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:54:52,006 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:54:52,009 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 17:54:52,015 ERROR][11][wan-tasty-kitalpha][T_1] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:54:52,015 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:54:55,066 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 9 -> 64 [2023-09-07 17:54:56,649 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109288, schain_finish_ts: 1692377364 [2023-09-07 17:54:57,751 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:54:57,751 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:54:57,751 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:54:57,751 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:54:57,751 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:54:57,752 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:54:57,753 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:54:57,753 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 17:54:57,776 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:54:57,827 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109288, schain_finish_ts: 1692637668 [2023-09-07 17:54:57,966 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:58,039 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:54:58,117 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:58,215 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:54:58,216 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2532 [2023-09-07 17:54:58,221 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:54:58,221 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:54:58,221 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:54:58,221 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:54:58,222 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:54:58,228 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:54:58,230 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:54:58.230295 [2023-09-07 17:54:58,375 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:58,402 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:54:58,434 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:54:58,443 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:54:58,451 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:54:58,451 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109298.4006448, 'te': 1694109298.4071429, 'initial_status': True} ====================================================================================================  [2023-09-07 17:54:58,453 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:54:58.453583 [2023-09-07 17:54:58,458 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:54:58,459 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:54:58,486 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109288, schain_finish_ts: 1693412520 [2023-09-07 17:54:58,946 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:54:58,946 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:54:58,946 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:54:58,946 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:54:58,947 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:54:58,947 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:54:58,948 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:54:58,949 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:54:58,972 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:54:59,294 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:59,447 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:59,524 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:54:59,524 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:54:59,526 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:54:59.526228 [2023-09-07 17:54:59,537 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:54:59,538 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:54:59,543 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:54:59,545 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:54:59,548 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2548 [2023-09-07 17:54:59,548 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:54:59,549 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:54:59,549 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:54:59,549 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:54:59,550 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:54:59,550 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109299.5357313, 'te': 1694109299.5499487, 'initial_status': True} ====================================================================================================  [2023-09-07 17:54:59,551 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:54:59.551850 [2023-09-07 17:54:59,556 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:54:59,556 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:54:59,556 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:54:59,558 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:54:59.558367 [2023-09-07 17:54:59,606 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:54:59,606 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:54:59,606 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:54:59,607 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:54:59,607 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:54:59,607 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:54:59,608 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:54:59,609 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 17:54:59,635 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:54:59,729 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:54:59,755 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:54:59,786 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:54:59,788 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:54:59,800 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:54:59,801 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109299.7539976, 'te': 1694109299.7600787, 'initial_status': True} ====================================================================================================  [2023-09-07 17:54:59,802 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:54:59.802805 [2023-09-07 17:54:59,807 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:54:59,808 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:54:59,871 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:00,048 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 35 -> 16 [2023-09-07 17:55:00,051 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:00,199 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:55:00,210 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2550 [2023-09-07 17:55:00,216 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:55:00,216 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:55:00,217 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:55:00,217 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:55:00,217 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:55:00,223 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:55:00,225 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:55:00.225640 [2023-09-07 17:55:00,407 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:00,441 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:55:00,602 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:55:00,604 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:55:00,612 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:55:00,612 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109300.4390862, 'te': 1694109300.4478085, 'initial_status': True} ====================================================================================================  [2023-09-07 17:55:00,614 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:55:00.614793 [2023-09-07 17:55:00,619 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:55:00,619 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:55:00,690 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109300, schain_finish_ts: 0 [2023-09-07 17:55:00,705 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:55:00,705 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:55:00,707 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:55:00.707800 [2023-09-07 17:55:00,715 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:55:00,715 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:55:00,722 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:55:00,727 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109300.714022, 'te': 1694109300.7270055, 'initial_status': True} ====================================================================================================  [2023-09-07 17:55:00,729 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:55:00.729494 [2023-09-07 17:55:00,734 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:55:00,734 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:55:01,497 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:55:01,497 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:55:01,500 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:55:01.500233 [2023-09-07 17:55:01,510 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:55:01,510 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:55:01,516 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:55:01,523 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109301.5087717, 'te': 1694109301.5228317, 'initial_status': True} ====================================================================================================  [2023-09-07 17:55:01,527 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:55:01.527756 [2023-09-07 17:55:01,535 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:55:01,536 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:55:01,831 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:55:01,831 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:55:01,832 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:55:02,204 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 17:55:02,204 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 17:55:05,450 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 55 -> 63 [2023-09-07 17:55:06,790 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109300, schain_finish_ts: 1694108304 [2023-09-07 17:55:07,936 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:55:07,937 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:55:07,937 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:55:11,014 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109300, schain_finish_ts: 1693742004 [2023-09-07 17:55:11,112 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 14 -> 11 [2023-09-07 17:55:12,140 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:55:12,141 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:55:12,141 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:55:12,142 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:55:12,142 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:55:12,142 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:55:12,164 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:55:12,338 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109300, schain_finish_ts: 1693742304 [2023-09-07 17:55:12,411 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:12,577 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:12,702 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:55:12,703 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2546 [2023-09-07 17:55:12,704 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:55:12,704 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:55:12,704 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:55:12,704 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:55:12,705 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:55:12,711 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:55:12,713 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:55:12.713078 [2023-09-07 17:55:12,869 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:12,901 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:55:12,931 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:55:12,933 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:55:12,940 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:55:12,941 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109312.8986878, 'te': 1694109312.9061103, 'initial_status': True} ====================================================================================================  [2023-09-07 17:55:12,942 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:55:12.942925 [2023-09-07 17:55:12,947 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:55:12,949 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:55:13,481 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:55:13,482 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:55:13,482 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:55:13,482 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 17:55:13,482 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 17:55:13,483 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:55:13,485 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:55:13,487 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 17:55:13,517 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:55:13,758 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:13,993 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:14,149 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:55:14,155 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2553 [2023-09-07 17:55:14,156 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:55:14,156 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:55:14,156 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:55:14,157 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:55:14,157 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:55:14,164 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:55:14,166 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:55:14.166794 [2023-09-07 17:55:14,334 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:14,360 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:55:14,394 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:55:14,396 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:55:14,404 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:55:14,404 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109314.3583903, 'te': 1694109314.365347, 'initial_status': True} ====================================================================================================  [2023-09-07 17:55:14,406 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:55:14.406357 [2023-09-07 17:55:14,411 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:55:14,412 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:55:15,290 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:55:15,291 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:55:15,292 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:55:15.292947 [2023-09-07 17:55:15,298 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:55:17,137 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 56 -> 60 [2023-09-07 17:55:21,835 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 17:55:22,010 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:55:22,010 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:55:22,011 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:55:22,011 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:55:22,191 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:55:22,191 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:55:22,193 INFO][8][curly-red-alterf][T_0] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 17:55:22,201 ERROR][8][curly-red-alterf][T_0] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:55:22,201 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:55:23,218 ERROR][24912][submissive-well-off-rotanev][T_0] - core.schains.task:26 - Task submissive-well-off-rotanev-config failed with max() arg is an empty sequence Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 63, in execute self.am.upstream_config() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 194, in upstream_config new_config = create_new_upstream_config( File "/usr/src/admin/core/schains/config/main.py", line 51, in create_new_upstream_config schain_config = generate_schain_config_with_skale( File "/usr/src/admin/core/schains/config/generator.py", line 213, in generate_schain_config_with_skale node_groups = get_previous_schain_groups(skale, schain_name) File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 56, in get_previous_schain_groups _add_previous_schain_rotations_state( File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 119, in _add_previous_schain_rotations_state new_node_id = max(previous_nodes.items(), key=lambda x: x[1]['finish_ts'])[0] ValueError: max() arg is an empty sequence [2023-09-07 17:55:23,218 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:55:27,572 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109324, schain_finish_ts: 1693412520 [2023-09-07 17:55:28,707 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:55:28,707 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:55:28,707 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:55:28,708 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:55:28,708 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:55:28,708 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:55:28,710 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:55:28,711 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 17:55:28,740 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:55:28,936 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:28,969 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:55:29,110 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:29,227 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:55:29,228 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2551 [2023-09-07 17:55:29,229 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:55:29,229 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:55:29,230 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:55:29,230 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:55:29,231 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:55:29,242 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:55:29,244 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:55:29.244841 [2023-09-07 17:55:29,439 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:29,463 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:55:29,495 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:55:29,496 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:55:29,505 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:55:29,505 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109329.4617343, 'te': 1694109329.4680457, 'initial_status': True} ====================================================================================================  [2023-09-07 17:55:29,507 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:55:29.507494 [2023-09-07 17:55:29,512 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:55:29,513 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:55:30,477 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:55:30,478 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:55:30,480 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:55:30.480072 [2023-09-07 17:55:30,486 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:55:30,486 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:55:30,492 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:55:30,496 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109330.4851708, 'te': 1694109330.4963162, 'initial_status': True} ====================================================================================================  [2023-09-07 17:55:30,498 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:55:30.498154 [2023-09-07 17:55:30,502 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:55:30,502 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:55:31,892 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109324, schain_finish_ts: 1692637668 [2023-09-07 17:55:33,024 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:55:33,024 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:55:33,024 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:55:33,024 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:55:33,025 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:55:33,025 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:55:33,026 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:55:33,029 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 17:55:33,053 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:55:33,241 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:33,398 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:33,529 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:55:33,531 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2549 [2023-09-07 17:55:33,532 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:55:33,532 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:55:33,532 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:55:33,533 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:55:33,533 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:55:33,540 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:55:33,544 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:55:33.544253 [2023-09-07 17:55:33,925 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:33,950 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:55:33,983 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:55:33,985 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:55:33,992 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:55:33,992 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109333.949236, 'te': 1694109333.955935, 'initial_status': True} ====================================================================================================  [2023-09-07 17:55:33,994 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:55:33.994256 [2023-09-07 17:55:33,998 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:55:33,998 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:55:34,427 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 17:55:34,428 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 17:55:34,707 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109324, schain_finish_ts: 1692377364 [2023-09-07 17:55:34,885 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:55:34,886 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:55:34,888 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:55:34.888267 [2023-09-07 17:55:34,898 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:55:34,898 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:55:34,906 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:55:34,912 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109334.8966105, 'te': 1694109334.9116893, 'initial_status': True} ====================================================================================================  [2023-09-07 17:55:34,914 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:55:34.914471 [2023-09-07 17:55:34,919 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:55:34,919 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:55:35,849 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:55:35,849 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:55:35,852 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:55:35,852 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:55:35,853 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:55:35,853 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:55:35,854 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:55:35,862 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 17:55:35,888 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:55:36,295 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:36,522 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:36,626 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:55:36,627 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2533 [2023-09-07 17:55:36,627 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:55:36,628 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:55:36,628 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:55:36,628 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:55:36,629 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:55:36,635 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:55:36,637 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:55:36.637063 [2023-09-07 17:55:36,790 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:36,819 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:55:36,849 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:55:36,851 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:55:36,859 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:55:36,860 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109336.8181908, 'te': 1694109336.8244324, 'initial_status': True} ====================================================================================================  [2023-09-07 17:55:36,861 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:55:36.861718 [2023-09-07 17:55:36,866 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:55:36,871 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:55:36,933 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109324, schain_finish_ts: 1694108304 [2023-09-07 17:55:37,680 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:55:37,681 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:55:37,682 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:55:37.682785 [2023-09-07 17:55:37,689 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:55:37,690 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:55:37,695 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:55:37,700 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109337.6885674, 'te': 1694109337.6999454, 'initial_status': True} ====================================================================================================  [2023-09-07 17:55:37,701 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:55:37.701824 [2023-09-07 17:55:37,706 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:55:37,706 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:55:37,785 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109336, schain_finish_ts: 0 [2023-09-07 17:55:38,059 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:55:38,059 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:55:38,060 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 17:55:38,930 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:55:38,931 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:55:38,931 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:40 - Task submissive-well-off-rotanev-config finished with None [2023-09-07 17:55:38,931 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:42 - Running task submissive-well-off-rotanev-config [2023-09-07 17:55:38,931 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 17:55:39,419 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109336, schain_finish_ts: 1693742304 [2023-09-07 17:55:40,544 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:55:40,544 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:55:40,544 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:55:40,544 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:55:40,545 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:55:40,545 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 17:55:40,563 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:55:40,640 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': False} [2023-09-07 17:55:40,640 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:55:40,649 INFO][24912][submissive-well-off-rotanev][T_0] - web.models.schain:152 - Changing monitor_last_seen for submissive-well-off-rotanev to 2023-09-07 17:55:40.649623 [2023-09-07 17:55:40,656 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.action:190 - Creating new upstream_config rotation_id: 8, stream: regression-2.2 [2023-09-07 17:55:40,662 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.config.main:49 - Generating sChain config for submissive-well-off-rotanev [2023-09-07 17:55:40,759 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:41,094 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:41,227 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:55:41,229 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2554 [2023-09-07 17:55:41,230 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:55:41,230 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:55:41,230 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:55:41,230 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:55:41,231 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:55:41,239 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:55:41,241 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:55:41.241895 [2023-09-07 17:55:41,408 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:41,433 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:55:41,460 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:55:41,467 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:55:41,475 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:55:41,475 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109341.4316647, 'te': 1694109341.4379587, 'initial_status': True} ====================================================================================================  [2023-09-07 17:55:41,477 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:55:41.477134 [2023-09-07 17:55:41,481 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:55:41,482 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:55:48,101 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109336, schain_finish_ts: 1693742004 [2023-09-07 17:55:49,292 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:55:49,293 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:55:49,293 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:55:49,293 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 17:55:49,293 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 17:55:49,293 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:55:49,296 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:55:49,300 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:55:49,327 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:55:49,668 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109336, schain_finish_ts: 1693412520 [2023-09-07 17:55:49,711 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:49,925 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:50,081 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:55:50,082 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2547 [2023-09-07 17:55:50,083 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:55:50,083 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:55:50,084 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:55:50,084 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:55:50,085 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:55:50,095 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:55:50,108 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:55:50.108174 [2023-09-07 17:55:50,296 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:50,327 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:55:50,380 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:55:50,386 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:55:50,394 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:55:50,394 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109350.3251402, 'te': 1694109350.3326783, 'initial_status': True} ====================================================================================================  [2023-09-07 17:55:50,396 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:55:50.396304 [2023-09-07 17:55:50,400 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:55:50,401 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:55:50,857 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:55:50,857 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:55:50,857 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:55:50,858 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:55:50,858 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:55:50,858 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:55:50,860 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:55:50,863 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 17:55:50,893 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:55:51,181 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:51,191 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:55:51,191 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:55:51,193 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:55:51.193867 [2023-09-07 17:55:51,198 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:55:51,344 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:51,479 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:55:51,481 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2552 [2023-09-07 17:55:51,481 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:55:51,482 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:55:51,482 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:55:51,482 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:55:51,483 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:55:51,489 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:55:51,492 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:55:51.492060 [2023-09-07 17:55:51,862 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:51,911 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:55:51,939 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:55:51,940 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:55:51,948 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:55:51,949 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109351.90206, 'te': 1694109351.9165742, 'initial_status': True} ====================================================================================================  [2023-09-07 17:55:51,950 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:55:51.950812 [2023-09-07 17:55:51,955 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:55:51,955 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:55:52,725 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:55:52,725 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:55:52,728 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:55:52.728795 [2023-09-07 17:55:52,736 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:55:52,737 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:55:52,745 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:55:52,751 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109352.7345393, 'te': 1694109352.7510965, 'initial_status': True} ====================================================================================================  [2023-09-07 17:55:52,753 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:55:52.753950 [2023-09-07 17:55:52,758 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:55:52,758 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:55:53,950 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 17:55:54,142 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:55:54,143 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:55:54,143 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:55:54,143 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:55:54,322 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:55:54,322 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:55:54,325 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 17:55:54,334 ERROR][11][wan-tasty-kitalpha][T_0] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:55:54,335 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:55:57,793 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109348, schain_finish_ts: 1692377364 [2023-09-07 17:55:58,913 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:55:58,913 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:55:58,913 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:55:58,914 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:55:58,914 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:55:58,914 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:55:58,916 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:55:58,916 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 17:55:58,944 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:55:59,175 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:59,389 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:59,526 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:55:59,528 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2534 [2023-09-07 17:55:59,529 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:55:59,529 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:55:59,530 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:55:59,531 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:55:59,531 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:55:59,544 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:55:59,552 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:55:59.552150 [2023-09-07 17:55:59,854 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:55:59,889 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:55:59,938 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:55:59,939 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:55:59,946 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:55:59,947 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109359.8879912, 'te': 1694109359.8942752, 'initial_status': True} ====================================================================================================  [2023-09-07 17:55:59,948 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:55:59.948779 [2023-09-07 17:55:59,954 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:55:59,955 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:56:00,032 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:56:00,778 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:56:00,778 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:56:00,780 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:56:00.780324 [2023-09-07 17:56:00,789 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:56:00,790 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:56:00,796 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:56:00,811 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109360.7884061, 'te': 1694109360.8110335, 'initial_status': True} ====================================================================================================  [2023-09-07 17:56:00,813 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:56:00.813411 [2023-09-07 17:56:00,835 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:56:00,836 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:56:02,482 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109360, schain_finish_ts: 1693742304 [2023-09-07 17:56:03,613 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:56:03,613 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:56:03,613 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:56:03,613 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 17:56:03,614 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 17:56:03,614 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:56:03,615 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:56:03,616 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 17:56:03,643 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:56:03,919 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:04,080 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:04,199 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:56:04,200 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2555 [2023-09-07 17:56:04,201 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:56:04,201 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:56:04,201 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:56:04,202 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:56:04,202 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:56:04,211 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:56:04,213 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:56:04.213475 [2023-09-07 17:56:04,459 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:04,493 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:56:04,523 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:56:04,525 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:56:04,532 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:56:04,533 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109364.4912906, 'te': 1694109364.4981377, 'initial_status': True} ====================================================================================================  [2023-09-07 17:56:04,534 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:56:04.534851 [2023-09-07 17:56:04,539 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:56:04,539 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:56:05,414 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:56:05,415 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:56:05,417 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:56:05.417044 [2023-09-07 17:56:05,421 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:56:05,894 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109360, schain_finish_ts: 0 [2023-09-07 17:56:07,051 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:56:07,052 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:56:07,052 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 17:56:07,131 INFO][1][MainProcess][MainThread] - core.schains.process_manager:57 - Process manager started [2023-09-07 17:56:07,343 INFO][1][MainProcess][MainThread] - core.schains.notifications:35 - Node account has 406903463189434189 WEI [2023-09-07 17:56:07,344 INFO][1][MainProcess][MainThread] - tools.notifications.messages:188 - Saving new balance state 3017 1 [2023-09-07 17:56:07,345 INFO][1][MainProcess][MainThread] - core.schains.process_manager:95 - Fetching schains to monitor... [2023-09-07 17:56:09,974 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109360, schain_finish_ts: 1692637668 [2023-09-07 17:56:10,200 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 17:56:10,201 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 17:56:11,130 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:56:11,131 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:56:11,131 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:56:11,131 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:56:11,132 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:56:11,132 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:56:11,133 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:56:11,135 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 17:56:11,159 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:56:11,456 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:11,625 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:11,725 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:56:11,726 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2550 [2023-09-07 17:56:11,727 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:56:11,727 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:56:11,728 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:56:11,728 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:56:11,729 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:56:11,734 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:56:11,737 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:56:11.737066 [2023-09-07 17:56:11,904 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:11,929 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:56:11,967 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:56:11,970 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:56:11,983 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:56:11,983 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109371.9282334, 'te': 1694109371.9342132, 'initial_status': True} ====================================================================================================  [2023-09-07 17:56:11,986 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:56:11.985988 [2023-09-07 17:56:11,990 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:56:11,991 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:56:12,016 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109360, schain_finish_ts: 1694108304 [2023-09-07 17:56:12,891 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:56:12,892 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:56:12,894 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:56:12.894293 [2023-09-07 17:56:12,902 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:56:12,903 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:56:12,909 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:56:12,914 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109372.901246, 'te': 1694109372.9141946, 'initial_status': True} ====================================================================================================  [2023-09-07 17:56:12,916 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:56:12.916071 [2023-09-07 17:56:12,920 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:56:12,921 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:56:12,957 INFO][1][MainProcess][MainThread] - core.schains.process_manager:109 - Get leaving_history for node ... [2023-09-07 17:56:13,154 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:56:13,154 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:56:13,155 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 17:56:14,924 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109372, schain_finish_ts: 1692377616 [2023-09-07 17:56:16,811 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109372, schain_finish_ts: 1693412520 [2023-09-07 17:56:16,926 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109372, schain_finish_ts: 0 [2023-09-07 17:56:17,296 INFO][1][MainProcess][MainThread] - core.schains.process_manager:117 - Got leaving sChains for the node: [] [2023-09-07 17:56:17,297 INFO][1][MainProcess][MainThread] - core.schains.process_manager:101 -  ==================================================================================================== Monitoring sChains Node ID: 16 sChains on node: [{'name': 'portly-passionate-sirius', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666281384, 'startBlock': 7803380, 'deposit': 0, 'index': 1, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x0000017723ca2', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'curly-red-alterf', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1676058936, 'startBlock': 8469646, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000001fe6cdfa', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'howling-hoedus', 'mainnetOwner': '0x1026467C91fdA40aCc7e6935d1E52f67274FD669', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666181400, 'startBlock': 7796660, 'deposit': 0, 'index': 0, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0xe5a600f4c2608', 'multitransactionMode': False, 'thresholdEncryption': False, 'active': True}, {'name': 'green-giddy-denebola', 'mainnetOwner': '0x8e835d5d9E28ffD259E8Fa8988Cacfc333A582B4', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674565164, 'startBlock': 8368049, 'deposit': 0, 'index': 4, 'generation': 1, 'originator': '0xd2001DAb6898127Be2F167B548691C87251D13C3', 'chainId': '0x00000585eb4b1', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'wan-tasty-kitalpha', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674568392, 'startBlock': 8368271, 'deposit': 0, 'index': 5, 'generation': 1, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x00000152e7be9', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'free-basic-rosy-SKALE', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1693996920, 'startBlock': 9646443, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000002bb90439', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'submissive-well-off-rotanev', 'mainnetOwner': '0x487360568fE5Ee4AE5c75782D5F63e9935B0E512', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1692361104, 'startBlock': 9539053, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000003791fa10', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}] Number of sChains on node: 7 Empty sChain structs: 0 ====================================================================================================  [2023-09-07 17:56:17,669 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:56:17,669 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain portly-passionate-sirius - Process is running: PID = 7 [2023-09-07 17:56:17,948 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:56:17,948 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:56:17,948 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:56:17,948 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:56:17,949 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:56:17,949 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:56:17,954 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:56:17,958 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:56:17,987 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:56:18,055 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:56:18,056 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain curly-red-alterf - Process is running: PID = 8 [2023-09-07 17:56:18,312 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:18,439 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:56:18,439 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain howling-hoedus - Process is running: PID = 9 [2023-09-07 17:56:18,468 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:18,589 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:56:18,590 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2553 [2023-09-07 17:56:18,591 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:56:18,591 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:56:18,591 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:56:18,592 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:56:18,592 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:56:18,599 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:56:18,601 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:56:18.601324 [2023-09-07 17:56:18,765 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:18,793 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:56:18,816 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:56:18,816 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain green-giddy-denebola - Process is running: PID = 10 [2023-09-07 17:56:18,831 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:56:18,833 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:56:18,842 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:56:18,842 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109378.791308, 'te': 1694109378.7994077, 'initial_status': True} ====================================================================================================  [2023-09-07 17:56:18,844 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:56:18.844438 [2023-09-07 17:56:18,849 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:56:18,849 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:56:19,197 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:56:19,197 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain wan-tasty-kitalpha - Process is running: PID = 11 [2023-09-07 17:56:19,576 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:56:19,576 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain free-basic-rosy-SKALE - Process is running: PID = 30622 [2023-09-07 17:56:19,766 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:56:19,766 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:56:19,768 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:56:19.768693 [2023-09-07 17:56:19,777 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:56:19,777 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:56:19,783 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:56:19,790 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109379.7759748, 'te': 1694109379.789833, 'initial_status': True} ====================================================================================================  [2023-09-07 17:56:19,791 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:56:19.791683 [2023-09-07 17:56:19,796 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:56:19,796 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:56:19,955 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:56:19,956 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain submissive-well-off-rotanev - Process is running: PID = 24912 [2023-09-07 17:56:19,956 INFO][1][MainProcess][MainThread] - core.schains.process_manager:88 - Process manager procedure finished [2023-09-07 17:56:19,956 INFO][1][MainProcess][MainThread] - __main__:65 - Sleeping for 90s after run_process_manager [2023-09-07 17:56:22,223 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109372, schain_finish_ts: 1693742004 [2023-09-07 17:56:23,347 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:56:23,347 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:56:23,347 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:56:23,348 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:56:23,348 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:56:23,348 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 17:56:23,371 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:56:23,610 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:23,811 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:23,940 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:56:23,942 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2548 [2023-09-07 17:56:23,943 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:56:23,944 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:56:23,944 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:56:23,945 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:56:23,945 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:56:23,962 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:56:23,964 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:56:23.964382 [2023-09-07 17:56:24,145 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:24,184 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:56:24,225 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:56:24,228 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:56:24,236 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:56:24,237 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109384.1814275, 'te': 1694109384.1907318, 'initial_status': True} ====================================================================================================  [2023-09-07 17:56:24,239 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:56:24.239289 [2023-09-07 17:56:24,245 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:56:24,248 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:56:24,376 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 17:56:24,376 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 17:56:25,859 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109372, schain_finish_ts: 1692377364 [2023-09-07 17:56:27,017 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:56:27,017 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:56:27,017 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:56:27,018 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:56:27,018 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:56:27,020 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:56:27,026 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:56:27,031 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 17:56:27,063 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:56:27,264 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:27,427 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:27,560 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:56:27,562 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2535 [2023-09-07 17:56:27,562 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:56:27,563 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:56:27,563 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:56:27,563 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:56:27,564 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:56:27,573 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:56:27,576 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:56:27.576834 [2023-09-07 17:56:27,756 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:27,786 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:56:27,815 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:56:27,817 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:56:27,827 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:56:27,827 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109387.7843125, 'te': 1694109387.792047, 'initial_status': True} ====================================================================================================  [2023-09-07 17:56:27,829 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:56:27.829953 [2023-09-07 17:56:27,836 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:56:27,837 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:56:28,874 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:56:28,875 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:56:28,881 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:56:28.881646 [2023-09-07 17:56:28,906 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:56:28,908 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:56:28,916 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:56:28,923 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109388.8946142, 'te': 1694109388.9223676, 'initial_status': True} ====================================================================================================  [2023-09-07 17:56:28,931 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:56:28.931253 [2023-09-07 17:56:28,938 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:56:28,938 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:56:29,547 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 17:56:29,712 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:56:29,713 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:56:29,713 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:56:29,713 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:56:29,898 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:56:29,898 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:56:29,901 INFO][8][curly-red-alterf][T_1] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 17:56:29,910 ERROR][8][curly-red-alterf][T_1] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:56:29,911 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:56:30,581 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109384, schain_finish_ts: 1693742304 [2023-09-07 17:56:30,987 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109384, schain_finish_ts: 0 [2023-09-07 17:56:31,001 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:56:31,714 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:56:31,715 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:56:31,716 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:56:31,716 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:56:31,716 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:56:31,716 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:56:31,752 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:56:31,983 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:32,154 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:56:32,155 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:56:32,155 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 17:56:32,273 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:32,397 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:56:32,398 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2556 [2023-09-07 17:56:32,399 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:56:32,399 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:56:32,399 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:56:32,399 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:56:32,400 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:56:32,406 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:56:32,408 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:56:32.408336 [2023-09-07 17:56:32,553 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:32,579 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:56:32,617 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:56:32,619 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:56:32,627 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:56:32,627 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109392.576591, 'te': 1694109392.584891, 'initial_status': True} ====================================================================================================  [2023-09-07 17:56:32,629 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:56:32.629666 [2023-09-07 17:56:32,634 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:56:32,635 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:56:33,069 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109384, schain_finish_ts: 1692637668 [2023-09-07 17:56:34,186 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:56:34,186 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:56:34,186 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:56:34,187 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:56:34,187 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:56:34,187 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:56:34,188 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:56:34,190 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:56:34,213 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:56:34,395 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:34,543 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:34,640 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:56:34,646 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2551 [2023-09-07 17:56:34,647 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:56:34,647 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:56:34,647 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:56:34,647 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:56:34,648 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:56:34,654 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:56:34,656 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:56:34.656017 [2023-09-07 17:56:34,976 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:35,003 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:56:35,036 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:56:35,038 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:56:35,050 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:56:35,051 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109395.001003, 'te': 1694109395.0082512, 'initial_status': True} ====================================================================================================  [2023-09-07 17:56:35,053 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:56:35.053617 [2023-09-07 17:56:35,058 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:56:35,059 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:56:36,025 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:56:36,026 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:56:36,028 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:56:36.028433 [2023-09-07 17:56:36,038 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:56:36,038 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:56:36,046 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:56:36,051 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109396.0364811, 'te': 1694109396.051252, 'initial_status': True} ====================================================================================================  [2023-09-07 17:56:36,053 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:56:36.053646 [2023-09-07 17:56:36,058 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:56:36,059 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:56:43,933 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 17:56:44,119 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:56:44,119 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:56:44,120 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:56:44,120 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:56:44,293 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109384, schain_finish_ts: 1693742004 [2023-09-07 17:56:44,300 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:56:44,300 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:56:44,302 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 17:56:44,307 ERROR][11][wan-tasty-kitalpha][T_1] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:56:44,308 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:56:45,419 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:56:45,419 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:56:45,420 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:56:45,420 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 17:56:45,420 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 17:56:45,421 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:56:45,427 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:56:45,432 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:56:45,452 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:56:45,780 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:45,928 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:46,030 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:56:46,031 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2549 [2023-09-07 17:56:46,031 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:56:46,031 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:56:46,032 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:56:46,032 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:56:46,032 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:56:46,038 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:56:46,040 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:56:46.040862 [2023-09-07 17:56:46,095 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109384, schain_finish_ts: 1694108304 [2023-09-07 17:56:46,196 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:46,231 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:56:46,261 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:56:46,263 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:56:46,270 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:56:46,271 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109406.2295883, 'te': 1694109406.2367408, 'initial_status': True} ====================================================================================================  [2023-09-07 17:56:46,272 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:56:46.272783 [2023-09-07 17:56:46,277 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:56:46,277 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:56:47,188 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:56:47,188 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:56:47,190 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:56:47.190786 [2023-09-07 17:56:47,199 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:56:47,242 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:56:47,245 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:56:47,246 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:56:47,956 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109384, schain_finish_ts: 1692377364 [2023-09-07 17:56:49,103 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:56:49,103 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:56:49,103 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:56:49,103 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:56:49,104 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:56:49,104 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:56:49,105 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:56:49,107 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 17:56:49,131 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:56:49,360 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:49,636 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:49,768 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:56:49,769 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2536 [2023-09-07 17:56:49,770 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:56:49,770 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:56:49,770 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:56:49,770 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:56:49,771 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:56:49,780 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:56:49,782 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:56:49.782918 [2023-09-07 17:56:50,203 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:50,238 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:56:50,301 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:56:50,308 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:56:50,325 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:56:50,327 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109410.235964, 'te': 1694109410.2443666, 'initial_status': True} ====================================================================================================  [2023-09-07 17:56:50,334 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:56:50.334302 [2023-09-07 17:56:50,343 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:56:50,344 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:56:50,986 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:56:50,987 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:56:50,989 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:56:50.989343 [2023-09-07 17:56:50,996 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:56:50,996 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:56:51,003 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:56:51,009 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109410.9946656, 'te': 1694109411.009304, 'initial_status': True} ====================================================================================================  [2023-09-07 17:56:51,011 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:56:51.011499 [2023-09-07 17:56:51,016 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:56:51,016 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:56:53,903 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109408, schain_finish_ts: 1693412520 [2023-09-07 17:56:55,046 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:56:55,046 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:56:55,047 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:56:55,047 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:56:55,047 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:56:55,048 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:56:55,048 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:56:55,048 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:56:55,086 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:56:55,422 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:55,681 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:55,812 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:56:55,813 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2554 [2023-09-07 17:56:55,815 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:56:55,815 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:56:55,815 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:56:55,816 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:56:55,817 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:56:55,825 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:56:55,831 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:56:55.830999 [2023-09-07 17:56:55,984 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:56:56,013 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:56:56,050 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:56:56,052 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:56:56,059 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:56:56,060 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109416.0116897, 'te': 1694109416.0180323, 'initial_status': True} ====================================================================================================  [2023-09-07 17:56:56,061 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:56:56.061533 [2023-09-07 17:56:56,066 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:56:56,067 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:56:56,839 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:56:56,839 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:56:56,841 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:56:56.841608 [2023-09-07 17:56:56,849 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:56:56,849 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:56:56,857 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:56:56,862 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109416.8476417, 'te': 1694109416.8626795, 'initial_status': True} ====================================================================================================  [2023-09-07 17:56:56,864 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:56:56.864763 [2023-09-07 17:56:56,870 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:56:56,870 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:57:00,098 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109408, schain_finish_ts: 0 [2023-09-07 17:57:01,248 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:57:01,248 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:57:01,248 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:57:01,932 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:57:06,243 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 17:57:06,244 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 17:57:10,370 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109420, schain_finish_ts: 1693742004 [2023-09-07 17:57:11,508 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:57:11,509 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:57:11,509 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:57:11,509 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:57:11,509 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:57:11,510 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 17:57:11,529 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:57:11,653 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109420, schain_finish_ts: 1693742304 [2023-09-07 17:57:11,718 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:11,866 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:11,968 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:57:11,969 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2550 [2023-09-07 17:57:11,970 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:57:11,971 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:57:11,971 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:57:11,971 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:57:11,972 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:57:11,979 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:57:11,981 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:57:11.981167 [2023-09-07 17:57:12,253 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:12,277 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:57:12,308 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:57:12,310 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:57:12,318 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:57:12,318 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109432.2759337, 'te': 1694109432.2832208, 'initial_status': True} ====================================================================================================  [2023-09-07 17:57:12,320 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:57:12.320061 [2023-09-07 17:57:12,327 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:57:12,327 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:57:12,786 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:57:12,787 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:57:12,787 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:57:12,787 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 17:57:12,787 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 17:57:12,788 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:57:12,790 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:57:12,793 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 17:57:12,822 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:57:13,061 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:13,240 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:13,367 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:57:13,368 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2557 [2023-09-07 17:57:13,369 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:57:13,369 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:57:13,369 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:57:13,370 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:57:13,370 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:57:13,377 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:57:13,381 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:57:13.381767 [2023-09-07 17:57:13,580 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:13,610 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:57:13,649 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:57:13,652 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:57:13,663 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:57:13,663 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109433.6088328, 'te': 1694109433.6160638, 'initial_status': True} ====================================================================================================  [2023-09-07 17:57:13,666 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:57:13.666581 [2023-09-07 17:57:13,680 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:57:13,682 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:57:14,157 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109432, schain_finish_ts: 1692637668 [2023-09-07 17:57:14,590 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:57:14,590 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:57:14,591 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:57:14.591973 [2023-09-07 17:57:14,597 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:57:15,060 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109432, schain_finish_ts: 1692377364 [2023-09-07 17:57:15,297 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:57:15,298 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:57:15,298 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:57:15,298 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:57:15,298 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:57:15,299 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:57:15,300 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:57:15,301 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:57:15,328 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:57:15,526 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:15,676 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:15,776 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:57:15,777 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2552 [2023-09-07 17:57:15,777 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:57:15,778 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:57:15,778 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:57:15,778 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:57:15,779 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:57:15,784 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:57:15,786 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:57:15.786688 [2023-09-07 17:57:15,948 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:15,976 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:57:16,017 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:57:16,019 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:57:16,026 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:57:16,026 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109435.9747863, 'te': 1694109435.98127, 'initial_status': True} ====================================================================================================  [2023-09-07 17:57:16,028 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:57:16.028248 [2023-09-07 17:57:16,035 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:57:16,035 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:57:16,202 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:57:16,202 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:57:16,202 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:57:16,203 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:57:16,203 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:57:16,203 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:57:16,204 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:57:16,205 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 17:57:16,230 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:57:16,443 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:16,614 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:16,732 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:57:16,733 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2537 [2023-09-07 17:57:16,734 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:57:16,734 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:57:16,734 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:57:16,735 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:57:16,735 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:57:16,742 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:57:16,744 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:57:16.744201 [2023-09-07 17:57:16,932 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:16,962 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:57:16,995 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:57:16,997 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:57:17,004 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:57:17,004 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109436.9582264, 'te': 1694109436.9682386, 'initial_status': True} ====================================================================================================  [2023-09-07 17:57:17,006 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:57:17.006505 [2023-09-07 17:57:17,012 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:57:17,013 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:57:17,072 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:57:17,072 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:57:17,074 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:57:17.074773 [2023-09-07 17:57:17,081 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:57:17,081 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:57:17,087 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:57:17,092 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109437.0801144, 'te': 1694109437.092143, 'initial_status': True} ====================================================================================================  [2023-09-07 17:57:17,093 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:57:17.093980 [2023-09-07 17:57:17,098 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:57:17,098 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:57:17,990 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:57:17,991 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:57:17,993 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:57:17.993653 [2023-09-07 17:57:18,001 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:57:18,002 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:57:18,008 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:57:18,013 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109437.9998844, 'te': 1694109438.0132592, 'initial_status': True} ====================================================================================================  [2023-09-07 17:57:18,015 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:57:18.015769 [2023-09-07 17:57:18,021 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:57:18,021 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:57:23,197 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109432, schain_finish_ts: 1694108304 [2023-09-07 17:57:24,328 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:57:24,328 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:57:24,329 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 17:57:25,669 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 17:57:25,854 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:57:25,854 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:57:25,855 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:57:25,855 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:57:26,032 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:57:26,033 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:57:26,036 INFO][8][curly-red-alterf][T_0] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 17:57:26,043 ERROR][8][curly-red-alterf][T_0] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:57:26,044 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:57:31,979 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:40 - Collecting rotation history for submissive-well-off-rotanev... [2023-09-07 17:57:32,870 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:57:33,105 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:50 - Rotation data for submissive-well-off-rotanev: Rotation(leaving_node_id=65, new_node_id=65, freeze_until=1694108100, rotation_counter=8) [2023-09-07 17:57:33,779 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 17:57:33,780 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 17:57:34,982 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109444, schain_finish_ts: 1693412520 [2023-09-07 17:57:36,087 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:57:36,088 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:57:36,088 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:57:36,089 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:57:36,089 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:57:36,089 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:57:36,093 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:57:36,095 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 17:57:36,121 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:57:36,309 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:36,468 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:36,603 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:57:36,605 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2555 [2023-09-07 17:57:36,605 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:57:36,605 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:57:36,606 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:57:36,606 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:57:36,606 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:57:36,613 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:57:36,619 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:57:36.619503 [2023-09-07 17:57:36,812 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:36,841 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:57:36,873 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:57:36,875 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:57:36,882 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:57:36,883 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109456.8393939, 'te': 1694109456.8489125, 'initial_status': True} ====================================================================================================  [2023-09-07 17:57:36,884 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:57:36.884786 [2023-09-07 17:57:36,889 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:57:36,890 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:57:37,189 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109444, schain_finish_ts: 0 [2023-09-07 17:57:37,448 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109456, schain_finish_ts: 1693742004 [2023-09-07 17:57:37,857 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:57:37,858 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:57:37,859 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:57:37.859954 [2023-09-07 17:57:37,866 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:57:37,866 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:57:37,872 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:57:37,877 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109457.8651445, 'te': 1694109457.8768258, 'initial_status': True} ====================================================================================================  [2023-09-07 17:57:37,878 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:57:37.878546 [2023-09-07 17:57:37,882 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:57:37,882 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:57:38,327 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:57:38,327 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:57:38,328 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 17:57:38,576 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:57:38,577 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:57:38,577 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:57:38,577 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 17:57:38,577 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 17:57:38,578 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:57:38,580 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:57:38,580 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 17:57:38,605 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:57:38,808 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:38,990 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:39,117 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:57:39,118 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2551 [2023-09-07 17:57:39,120 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:57:39,120 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:57:39,120 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:57:39,121 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:57:39,122 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:57:39,134 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:57:39,152 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:57:39.152152 [2023-09-07 17:57:39,313 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:39,339 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:57:39,367 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:57:39,369 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:57:39,376 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:57:39,377 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109459.3373263, 'te': 1694109459.3438394, 'initial_status': True} ====================================================================================================  [2023-09-07 17:57:39,378 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:57:39.378860 [2023-09-07 17:57:39,383 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:57:39,384 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:57:40,330 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:57:40,330 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:57:40,333 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:57:40.333308 [2023-09-07 17:57:40,337 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:57:46,161 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109456, schain_finish_ts: 1692377364 [2023-09-07 17:57:47,300 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:57:47,300 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:57:47,301 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:57:47,301 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:57:47,301 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:57:47,301 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:57:47,303 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:57:47,304 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 17:57:47,332 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:57:47,521 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:47,670 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:47,769 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:57:47,770 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2538 [2023-09-07 17:57:47,771 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:57:47,771 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:57:47,771 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:57:47,772 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:57:47,772 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:57:47,778 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:57:47,780 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:57:47.780864 [2023-09-07 17:57:48,046 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:48,070 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:57:48,096 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:57:48,098 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:57:48,105 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:57:48,105 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109468.0683582, 'te': 1694109468.0752242, 'initial_status': True} ====================================================================================================  [2023-09-07 17:57:48,109 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:57:48.109440 [2023-09-07 17:57:48,114 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:57:48,114 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:57:48,156 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 63 -> 65 [2023-09-07 17:57:49,053 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:57:49,053 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:57:49,055 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:57:49.055284 [2023-09-07 17:57:49,064 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:57:49,064 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:57:49,069 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:57:49,074 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109469.0627768, 'te': 1694109469.0742724, 'initial_status': True} ====================================================================================================  [2023-09-07 17:57:49,076 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:57:49.076372 [2023-09-07 17:57:49,081 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:57:49,081 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:57:49,694 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109456, schain_finish_ts: 1693742304 [2023-09-07 17:57:50,021 INFO][1][MainProcess][MainThread] - core.schains.cleaner:65 - Cleaner process started [2023-09-07 17:57:50,069 INFO][30915][cleaner][MainThread] - core.schains.cleaner:110 - Cleaner procedure started. [2023-09-07 17:57:50,807 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:57:50,809 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:57:50,809 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:57:50,809 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:57:50,809 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:57:50,810 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 17:57:50,830 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:57:51,186 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:51,354 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:51,466 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:57:51,467 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2558 [2023-09-07 17:57:51,468 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:57:51,468 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:57:51,468 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:57:51,468 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:57:51,469 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:57:51,475 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:57:51,477 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:57:51.477439 [2023-09-07 17:57:51,653 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:51,679 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:57:51,710 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:57:51,712 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:57:51,720 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:57:51,720 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109471.6774528, 'te': 1694109471.6843588, 'initial_status': True} ====================================================================================================  [2023-09-07 17:57:51,722 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:57:51.722615 [2023-09-07 17:57:51,726 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:57:51,727 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:57:53,010 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 23 -> 24 [2023-09-07 17:57:53,254 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109468, schain_finish_ts: 1692637668 [2023-09-07 17:57:53,327 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 17:57:53,515 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:57:53,520 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:57:53,521 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:57:53,522 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:57:53,696 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:57:53,696 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:57:53,699 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 17:57:53,704 ERROR][11][wan-tasty-kitalpha][T_0] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:57:53,704 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:57:54,408 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:57:54,408 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:57:54,409 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:57:54,409 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:57:54,409 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:57:54,409 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:57:54,411 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:57:54,414 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 17:57:54,442 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:57:54,667 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:54,931 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:55,048 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:57:55,057 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2553 [2023-09-07 17:57:55,058 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:57:55,058 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:57:55,059 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:57:55,059 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:57:55,059 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:57:55,065 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:57:55,067 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:57:55.067878 [2023-09-07 17:57:55,234 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:57:55,262 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:57:55,292 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:57:55,294 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:57:55,302 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:57:55,302 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109475.2602403, 'te': 1694109475.268184, 'initial_status': True} ====================================================================================================  [2023-09-07 17:57:55,304 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:57:55.304826 [2023-09-07 17:57:55,311 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:57:55,312 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:57:55,727 INFO][30915][cleaner][MainThread] - core.schains.cleaner:116 - sChains on contracts: ['portly-passionate-sirius', 'curly-red-alterf', 'howling-hoedus', 'green-giddy-denebola', 'wan-tasty-kitalpha', 'free-basic-rosy-SKALE', 'submissive-well-off-rotanev'] sChains on node: ['curly-red-alterf', 'free-basic-rosy-SKALE', 'green-giddy-denebola', 'howling-hoedus', 'portly-passionate-sirius', 'submissive-well-off-rotanev', 'wan-tasty-kitalpha'] [2023-09-07 17:57:55,728 INFO][30915][cleaner][MainThread] - core.schains.cleaner:132 - Cleanup procedure finished [2023-09-07 17:57:55,732 INFO][1][MainProcess][MainThread] - core.schains.cleaner:67 - Cleaner process is joined. [2023-09-07 17:57:55,732 INFO][1][MainProcess][MainThread] - core.schains.cleaner:68 - Terminating the process [2023-09-07 17:57:55,733 INFO][1][MainProcess][MainThread] - __main__:70 - Sleeping for 90s after run_cleaner [2023-09-07 17:57:56,218 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:57:56,218 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:57:56,220 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:57:56.220539 [2023-09-07 17:57:56,227 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:57:56,227 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:57:56,233 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:57:56,237 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109476.2259011, 'te': 1694109476.2375104, 'initial_status': True} ====================================================================================================  [2023-09-07 17:57:56,239 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:57:56.239743 [2023-09-07 17:57:56,244 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:57:56,244 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:57:57,483 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 9 -> 64 [2023-09-07 17:57:59,169 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 17:57:59,170 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 17:58:00,271 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109468, schain_finish_ts: 1694108304 [2023-09-07 17:58:01,385 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:58:01,386 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:58:01,386 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 17:58:02,296 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109480, schain_finish_ts: 0 [2023-09-07 17:58:02,457 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 35 -> 16 [2023-09-07 17:58:03,044 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109480, schain_finish_ts: 1693412520 [2023-09-07 17:58:03,455 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:58:03,455 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:58:03,456 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 17:58:03,795 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:58:04,180 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:58:04,180 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:58:04,180 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:58:04,181 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:58:04,181 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:58:04,181 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:58:04,183 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:58:04,183 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:58:04,220 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:58:04,615 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:04,836 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:04,962 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:58:04,963 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2556 [2023-09-07 17:58:04,964 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:58:04,964 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:58:04,965 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:58:04,966 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:58:04,967 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:58:04,987 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:58:04,990 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:58:04.990623 [2023-09-07 17:58:05,190 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:05,235 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:58:05,288 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:58:05,291 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:58:05,300 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:58:05,301 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109485.2322922, 'te': 1694109485.2418437, 'initial_status': True} ====================================================================================================  [2023-09-07 17:58:05,302 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:58:05.302557 [2023-09-07 17:58:05,307 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:58:05,307 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:58:06,066 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:58:06,066 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:58:06,070 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:58:06.070678 [2023-09-07 17:58:06,079 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:58:06,079 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:58:06,087 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:58:06,092 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109486.077445, 'te': 1694109486.0925407, 'initial_status': True} ====================================================================================================  [2023-09-07 17:58:06,094 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:58:06.094610 [2023-09-07 17:58:06,101 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:58:06,101 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:58:07,840 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 55 -> 63 [2023-09-07 17:58:12,740 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109480, schain_finish_ts: 1693742304 [2023-09-07 17:58:13,568 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 14 -> 11 [2023-09-07 17:58:13,864 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:58:13,865 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:58:13,866 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:58:13,872 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 17:58:13,872 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 17:58:13,872 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:58:13,874 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:58:13,875 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 17:58:13,907 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:58:14,137 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:14,333 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:14,470 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:58:14,471 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2559 [2023-09-07 17:58:14,472 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:58:14,472 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:58:14,472 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:58:14,472 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:58:14,473 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:58:14,487 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:58:14,500 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:58:14.500622 [2023-09-07 17:58:14,690 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:14,717 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:58:14,744 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:58:14,746 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:58:14,754 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:58:14,754 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109494.7155612, 'te': 1694109494.7228675, 'initial_status': True} ====================================================================================================  [2023-09-07 17:58:14,755 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:58:14.755975 [2023-09-07 17:58:14,760 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:58:14,761 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:58:15,537 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109492, schain_finish_ts: 1693742004 [2023-09-07 17:58:15,695 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:58:15,695 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:58:15,697 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:58:15.697423 [2023-09-07 17:58:15,703 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:58:16,671 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:58:16,671 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:58:16,671 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:58:16,671 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:58:16,671 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:58:16,672 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 17:58:16,703 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:58:16,943 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:17,078 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:17,174 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:58:17,175 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2552 [2023-09-07 17:58:17,176 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:58:17,176 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:58:17,177 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:58:17,177 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:58:17,178 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:58:17,184 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:58:17,186 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:58:17.186187 [2023-09-07 17:58:17,437 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:17,467 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:58:17,494 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:58:17,496 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:58:17,504 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:58:17,504 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109497.4644868, 'te': 1694109497.4726322, 'initial_status': True} ====================================================================================================  [2023-09-07 17:58:17,506 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:58:17.506555 [2023-09-07 17:58:17,514 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:58:17,514 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:58:18,586 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 17:58:18,765 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:58:18,765 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:58:18,765 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:58:18,766 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:58:18,933 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:58:18,934 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:58:18,936 INFO][8][curly-red-alterf][T_1] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 17:58:18,945 ERROR][8][curly-red-alterf][T_1] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:58:18,946 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:58:19,691 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 56 -> 60 [2023-09-07 17:58:25,263 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109492, schain_finish_ts: 1692377364 [2023-09-07 17:58:25,860 ERROR][24912][submissive-well-off-rotanev][T_0] - core.schains.task:26 - Task submissive-well-off-rotanev-config failed with max() arg is an empty sequence Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 63, in execute self.am.upstream_config() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 194, in upstream_config new_config = create_new_upstream_config( File "/usr/src/admin/core/schains/config/main.py", line 51, in create_new_upstream_config schain_config = generate_schain_config_with_skale( File "/usr/src/admin/core/schains/config/generator.py", line 213, in generate_schain_config_with_skale node_groups = get_previous_schain_groups(skale, schain_name) File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 56, in get_previous_schain_groups _add_previous_schain_rotations_state( File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 119, in _add_previous_schain_rotations_state new_node_id = max(previous_nodes.items(), key=lambda x: x[1]['finish_ts'])[0] ValueError: max() arg is an empty sequence [2023-09-07 17:58:25,861 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:58:26,411 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:58:26,412 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:58:26,412 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:58:26,412 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:58:26,413 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:58:26,413 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:58:26,415 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:58:26,416 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 17:58:26,447 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:58:26,687 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:26,892 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:27,011 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:58:27,012 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2539 [2023-09-07 17:58:27,013 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:58:27,013 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:58:27,014 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:58:27,014 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:58:27,015 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:58:27,021 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:58:27,023 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:58:27.023410 [2023-09-07 17:58:27,122 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109504, schain_finish_ts: 1693412520 [2023-09-07 17:58:27,205 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:27,235 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:58:27,266 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:58:27,268 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:58:27,279 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:58:27,280 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109507.2326086, 'te': 1694109507.240437, 'initial_status': True} ====================================================================================================  [2023-09-07 17:58:27,282 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:58:27.282248 [2023-09-07 17:58:27,286 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:58:27,287 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:58:28,173 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:58:28,173 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:58:28,175 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:58:28.175566 [2023-09-07 17:58:28,184 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:58:28,186 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:58:28,191 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:58:28,200 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109508.1811905, 'te': 1694109508.2003536, 'initial_status': True} ====================================================================================================  [2023-09-07 17:58:28,207 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:58:28.207945 [2023-09-07 17:58:28,214 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:58:28,214 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:58:28,265 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:58:28,265 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:58:28,265 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:58:28,266 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:58:28,266 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:58:28,266 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:58:28,268 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:58:28,271 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:58:28,298 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:58:28,359 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109504, schain_finish_ts: 1692637668 [2023-09-07 17:58:28,496 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:28,670 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:28,802 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:58:28,803 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2557 [2023-09-07 17:58:28,804 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:58:28,804 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:58:28,805 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:58:28,805 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:58:28,805 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:58:28,812 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:58:28,815 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:58:28.815101 [2023-09-07 17:58:28,984 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:29,009 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:58:29,040 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:58:29,042 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:58:29,052 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:58:29,053 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109509.007798, 'te': 1694109509.0152254, 'initial_status': True} ====================================================================================================  [2023-09-07 17:58:29,054 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:58:29.054733 [2023-09-07 17:58:29,059 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:58:29,060 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:58:29,497 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:58:29,497 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:58:29,497 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:58:29,498 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:58:29,498 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:58:29,498 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:58:29,500 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:58:29,500 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 17:58:29,528 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:58:29,735 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:29,889 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:30,012 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:58:30,013 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2554 [2023-09-07 17:58:30,014 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:58:30,014 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:58:30,014 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:58:30,015 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:58:30,015 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:58:30,021 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:58:30,023 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:58:30.022994 [2023-09-07 17:58:30,079 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:58:30,079 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:58:30,081 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:58:30.081767 [2023-09-07 17:58:30,089 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:58:30,089 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:58:30,095 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:58:30,101 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109510.0875428, 'te': 1694109510.1011329, 'initial_status': True} ====================================================================================================  [2023-09-07 17:58:30,103 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:58:30.103090 [2023-09-07 17:58:30,107 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:58:30,108 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:58:30,213 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:30,242 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:58:30,273 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:58:30,275 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:58:30,283 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:58:30,284 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109510.2406352, 'te': 1694109510.2477531, 'initial_status': True} ====================================================================================================  [2023-09-07 17:58:30,285 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:58:30.285795 [2023-09-07 17:58:30,290 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:58:30,290 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:58:31,265 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:58:31,266 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:58:31,268 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:58:31.268041 [2023-09-07 17:58:31,274 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:58:31,275 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:58:31,281 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:58:31,286 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109511.2735403, 'te': 1694109511.286509, 'initial_status': True} ====================================================================================================  [2023-09-07 17:58:31,289 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:58:31.289715 [2023-09-07 17:58:31,295 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:58:31,295 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:58:32,318 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109504, schain_finish_ts: 1694108304 [2023-09-07 17:58:33,444 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:58:33,444 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:58:33,445 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 17:58:34,600 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 17:58:34,600 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 17:58:34,734 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:58:37,839 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109504, schain_finish_ts: 1693742304 [2023-09-07 17:58:38,404 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109504, schain_finish_ts: 0 [2023-09-07 17:58:38,609 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109504, schain_finish_ts: 1693742004 [2023-09-07 17:58:38,979 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:58:38,979 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:58:38,979 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:58:38,980 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:58:38,980 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:58:38,980 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:58:39,001 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:58:39,334 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:39,482 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:39,531 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:58:39,531 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:58:39,531 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:40 - Task submissive-well-off-rotanev-config finished with None [2023-09-07 17:58:39,531 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:42 - Running task submissive-well-off-rotanev-config [2023-09-07 17:58:39,532 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 17:58:39,579 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:58:39,580 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2560 [2023-09-07 17:58:39,581 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:58:39,581 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:58:39,581 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:58:39,581 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:58:39,582 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:58:39,588 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:58:39,590 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:58:39.590425 [2023-09-07 17:58:39,744 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:39,753 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:58:39,753 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:58:39,753 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:58:39,754 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 17:58:39,754 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 17:58:39,754 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:58:39,756 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:58:39,757 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 17:58:39,772 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:58:39,782 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:58:39,808 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:58:39,810 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:58:39,818 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:58:39,818 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109519.7701566, 'te': 1694109519.7768836, 'initial_status': True} ====================================================================================================  [2023-09-07 17:58:39,820 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:58:39.820361 [2023-09-07 17:58:39,824 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:58:39,825 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:58:39,973 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:40,131 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:40,228 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:58:40,229 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2553 [2023-09-07 17:58:40,230 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:58:40,230 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:58:40,230 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:58:40,230 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:58:40,231 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:58:40,237 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:58:40,239 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:58:40.239257 [2023-09-07 17:58:40,387 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:40,417 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:58:40,473 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:58:40,475 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:58:40,483 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:58:40,483 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109520.4150703, 'te': 1694109520.4240727, 'initial_status': True} ====================================================================================================  [2023-09-07 17:58:40,485 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:58:40.485275 [2023-09-07 17:58:40,489 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:58:40,490 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:58:41,267 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': False} [2023-09-07 17:58:41,268 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:58:41,271 INFO][24912][submissive-well-off-rotanev][T_0] - web.models.schain:152 - Changing monitor_last_seen for submissive-well-off-rotanev to 2023-09-07 17:58:41.271097 [2023-09-07 17:58:41,278 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.action:190 - Creating new upstream_config rotation_id: 8, stream: regression-2.2 [2023-09-07 17:58:41,281 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.config.main:49 - Generating sChain config for submissive-well-off-rotanev [2023-09-07 17:58:41,568 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:58:41,568 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:58:41,572 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:58:41.572133 [2023-09-07 17:58:41,578 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:58:53,475 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109528, schain_finish_ts: 1692637668 [2023-09-07 17:58:53,925 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 17:58:54,104 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:58:54,105 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:58:54,105 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:58:54,105 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:58:54,277 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:58:54,277 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:58:54,280 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 17:58:54,286 ERROR][11][wan-tasty-kitalpha][T_1] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:58:54,287 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:58:54,622 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:58:54,622 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:58:54,622 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:58:54,623 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:58:54,623 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:58:54,623 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:58:54,627 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:58:54,628 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 17:58:54,648 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:58:54,864 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:55,132 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:55,235 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:58:55,236 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2555 [2023-09-07 17:58:55,237 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:58:55,237 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:58:55,237 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:58:55,237 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:58:55,238 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:58:55,243 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:58:55,245 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:58:55.245441 [2023-09-07 17:58:55,405 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:58:55,434 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:58:55,465 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:58:55,467 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:58:55,475 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:58:55,475 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109535.4286206, 'te': 1694109535.440346, 'initial_status': True} ====================================================================================================  [2023-09-07 17:58:55,477 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:58:55.477148 [2023-09-07 17:58:55,483 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:58:55,484 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:58:56,471 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:58:56,471 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:58:56,473 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:58:56.473195 [2023-09-07 17:58:56,479 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:58:56,479 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:58:56,485 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:58:56,490 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109536.478289, 'te': 1694109536.4900472, 'initial_status': True} ====================================================================================================  [2023-09-07 17:58:56,491 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:58:56.491864 [2023-09-07 17:58:56,496 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:58:56,496 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:58:57,386 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109528, schain_finish_ts: 1694108304 [2023-09-07 17:58:58,514 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:58:58,514 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:58:58,515 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:59:00,690 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 17:59:00,690 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 17:59:03,918 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109540, schain_finish_ts: 1693742304 [2023-09-07 17:59:05,041 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:59:05,041 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:59:05,041 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:59:05,042 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 17:59:05,042 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 17:59:05,042 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:59:05,044 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:59:05,046 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 17:59:05,069 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:59:05,360 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109540, schain_finish_ts: 1692377364 [2023-09-07 17:59:05,412 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:05,588 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:05,680 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:59:05,714 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:59:05,715 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2561 [2023-09-07 17:59:05,716 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:59:05,716 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:59:05,716 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:59:05,717 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:59:05,717 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:59:05,723 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:59:05,725 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:59:05.725514 [2023-09-07 17:59:05,908 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:05,935 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:59:05,966 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:59:05,968 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:59:05,974 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:59:05,975 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109545.9337714, 'te': 1694109545.9431245, 'initial_status': True} ====================================================================================================  [2023-09-07 17:59:05,977 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:59:05.977746 [2023-09-07 17:59:05,983 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:59:05,984 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:59:06,502 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:59:06,503 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:59:06,503 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:59:06,503 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:59:06,503 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:59:06,503 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:59:06,505 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:59:06,507 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 17:59:06,530 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:59:06,761 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:06,844 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:59:06,844 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:59:06,847 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:59:06.846984 [2023-09-07 17:59:06,852 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:59:07,081 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:07,194 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:59:07,195 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2540 [2023-09-07 17:59:07,196 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:59:07,196 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:59:07,197 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:59:07,197 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:59:07,198 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:59:07,206 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:59:07,209 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:59:07.209397 [2023-09-07 17:59:07,414 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:07,441 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:59:07,474 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:59:07,476 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:59:07,483 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:59:07,484 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109547.4383652, 'te': 1694109547.4452016, 'initial_status': True} ====================================================================================================  [2023-09-07 17:59:07,485 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:59:07.485420 [2023-09-07 17:59:07,489 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:59:07,496 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:59:08,219 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109540, schain_finish_ts: 1693412520 [2023-09-07 17:59:08,291 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:59:08,291 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:59:08,297 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:59:08.297915 [2023-09-07 17:59:08,304 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:59:08,304 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:59:08,312 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:59:08,317 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109548.3032916, 'te': 1694109548.3171709, 'initial_status': True} ====================================================================================================  [2023-09-07 17:59:08,320 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:59:08.320609 [2023-09-07 17:59:08,327 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:59:08,327 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:59:09,359 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:59:09,361 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:59:09,362 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:59:09,362 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:59:09,363 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:59:09,363 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:59:09,364 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:59:09,364 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:59:09,402 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:59:09,492 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109540, schain_finish_ts: 0 [2023-09-07 17:59:09,695 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109540, schain_finish_ts: 1693742004 [2023-09-07 17:59:09,749 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:09,903 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:10,008 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:59:10,013 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2558 [2023-09-07 17:59:10,013 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:59:10,013 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:59:10,014 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:59:10,014 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:59:10,014 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:59:10,020 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:59:10,022 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:59:10.022202 [2023-09-07 17:59:10,190 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:10,217 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:59:10,244 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:59:10,246 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:59:10,253 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:59:10,253 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109550.2148907, 'te': 1694109550.2217035, 'initial_status': True} ====================================================================================================  [2023-09-07 17:59:10,255 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:59:10.255108 [2023-09-07 17:59:10,259 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:59:10,260 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:59:10,649 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:59:10,649 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:59:10,649 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:59:10,819 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:59:10,820 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:59:10,820 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:59:10,820 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:59:10,820 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:59:10,820 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:59:10,839 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:59:11,051 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:11,191 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:59:11,191 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:59:11,193 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:59:11.193442 [2023-09-07 17:59:11,201 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:59:11,201 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:59:11,208 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:59:11,215 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109551.1995802, 'te': 1694109551.2154427, 'initial_status': True} ====================================================================================================  [2023-09-07 17:59:11,217 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:59:11.217319 [2023-09-07 17:59:11,223 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:59:11,223 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:59:11,313 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:11,418 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:59:11,419 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2554 [2023-09-07 17:59:11,420 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:59:11,420 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:59:11,421 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:59:11,421 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:59:11,421 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:59:11,428 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:59:11,430 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:59:11.430191 [2023-09-07 17:59:11,578 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:11,609 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:59:11,638 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:59:11,640 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:59:11,648 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:59:11,648 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109551.6055357, 'te': 1694109551.615521, 'initial_status': True} ====================================================================================================  [2023-09-07 17:59:11,650 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:59:11.650151 [2023-09-07 17:59:11,654 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:59:11,655 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:59:20,176 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 17:59:20,349 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:59:20,349 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:59:20,350 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 17:59:20,350 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:59:20,523 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:59:20,524 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:59:20,526 INFO][8][curly-red-alterf][T_0] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 17:59:20,530 ERROR][8][curly-red-alterf][T_0] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:59:20,530 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:59:21,468 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109552, schain_finish_ts: 1694108304 [2023-09-07 17:59:22,614 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:59:22,614 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:59:22,614 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 17:59:25,696 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 17:59:25,696 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 17:59:25,814 INFO][1][MainProcess][MainThread] - core.schains.process_manager:57 - Process manager started [2023-09-07 17:59:26,024 INFO][1][MainProcess][MainThread] - core.schains.notifications:35 - Node account has 406903463189434189 WEI [2023-09-07 17:59:26,026 INFO][1][MainProcess][MainThread] - tools.notifications.messages:188 - Saving new balance state 3018 1 [2023-09-07 17:59:26,026 INFO][1][MainProcess][MainThread] - core.schains.process_manager:95 - Fetching schains to monitor... [2023-09-07 17:59:27,406 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109552, schain_finish_ts: 1692377364 [2023-09-07 17:59:28,513 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:59:28,513 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:59:28,513 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:59:28,513 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:59:28,513 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:59:28,514 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:59:28,517 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:59:28,518 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:59:28,542 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:59:28,793 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:29,012 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:29,122 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:59:29,123 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2541 [2023-09-07 17:59:29,124 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:59:29,124 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:59:29,124 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:59:29,125 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:59:29,125 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:59:29,132 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:59:29,134 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:59:29.134440 [2023-09-07 17:59:29,420 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:29,453 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:59:29,513 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:59:29,515 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:59:29,523 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:59:29,524 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109569.4513075, 'te': 1694109569.4592624, 'initial_status': True} ====================================================================================================  [2023-09-07 17:59:29,526 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:59:29.526042 [2023-09-07 17:59:29,530 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:59:29,530 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:59:30,314 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:59:30,314 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:59:30,317 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:59:30.317073 [2023-09-07 17:59:30,326 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:59:30,327 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:59:30,335 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:59:30,342 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109570.3241343, 'te': 1694109570.3415654, 'initial_status': True} ====================================================================================================  [2023-09-07 17:59:30,344 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:59:30.344210 [2023-09-07 17:59:30,351 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:59:30,351 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:59:31,577 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109552, schain_finish_ts: 1692637668 [2023-09-07 17:59:31,615 INFO][1][MainProcess][MainThread] - core.schains.process_manager:109 - Get leaving_history for node ... [2023-09-07 17:59:32,723 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:59:32,723 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:59:32,724 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:59:32,724 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 17:59:32,724 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 17:59:32,725 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 17:59:32,727 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 17:59:32,729 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 17:59:32,751 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 17:59:32,959 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:33,104 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:33,204 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:59:33,205 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2556 [2023-09-07 17:59:33,206 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 17:59:33,206 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:59:33,206 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:59:33,207 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:59:33,207 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:59:33,214 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:59:33,216 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:59:33.215976 [2023-09-07 17:59:33,363 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:33,390 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 17:59:33,417 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 17:59:33,420 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 17:59:33,428 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:59:33,429 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109573.3887951, 'te': 1694109573.3957882, 'initial_status': True} ====================================================================================================  [2023-09-07 17:59:33,437 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:59:33.437104 [2023-09-07 17:59:33,441 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:59:33,442 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:59:33,589 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109552, schain_finish_ts: 0 [2023-09-07 17:59:33,643 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109552, schain_finish_ts: 1692377616 [2023-09-07 17:59:34,299 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109552, schain_finish_ts: 1693412520 [2023-09-07 17:59:34,506 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:59:34,507 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:59:34,510 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:59:34.510691 [2023-09-07 17:59:34,518 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 17:59:34,518 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 17:59:34,524 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 17:59:34,529 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109574.5160024, 'te': 1694109574.5290468, 'initial_status': True} ====================================================================================================  [2023-09-07 17:59:34,531 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 17:59:34.531131 [2023-09-07 17:59:34,535 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:59:34,536 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:59:34,725 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:59:34,725 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:59:34,726 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 17:59:35,420 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:59:35,421 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:59:35,421 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:59:35,421 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 17:59:35,421 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 17:59:35,422 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 17:59:35,423 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 17:59:35,424 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 17:59:35,447 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 17:59:35,592 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109552, schain_finish_ts: 0 [2023-09-07 17:59:35,637 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:35,782 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:35,889 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:59:35,890 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2559 [2023-09-07 17:59:35,891 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 17:59:35,891 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:59:35,891 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:59:35,891 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:59:35,892 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:59:35,897 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:59:35,899 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:59:35.899579 [2023-09-07 17:59:35,960 INFO][1][MainProcess][MainThread] - core.schains.process_manager:117 - Got leaving sChains for the node: [] [2023-09-07 17:59:35,961 INFO][1][MainProcess][MainThread] - core.schains.process_manager:101 -  ==================================================================================================== Monitoring sChains Node ID: 16 sChains on node: [{'name': 'portly-passionate-sirius', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666281384, 'startBlock': 7803380, 'deposit': 0, 'index': 1, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x0000017723ca2', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'curly-red-alterf', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1676058936, 'startBlock': 8469646, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000001fe6cdfa', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'howling-hoedus', 'mainnetOwner': '0x1026467C91fdA40aCc7e6935d1E52f67274FD669', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666181400, 'startBlock': 7796660, 'deposit': 0, 'index': 0, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0xe5a600f4c2608', 'multitransactionMode': False, 'thresholdEncryption': False, 'active': True}, {'name': 'green-giddy-denebola', 'mainnetOwner': '0x8e835d5d9E28ffD259E8Fa8988Cacfc333A582B4', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674565164, 'startBlock': 8368049, 'deposit': 0, 'index': 4, 'generation': 1, 'originator': '0xd2001DAb6898127Be2F167B548691C87251D13C3', 'chainId': '0x00000585eb4b1', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'wan-tasty-kitalpha', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674568392, 'startBlock': 8368271, 'deposit': 0, 'index': 5, 'generation': 1, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x00000152e7be9', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'free-basic-rosy-SKALE', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1693996920, 'startBlock': 9646443, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000002bb90439', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'submissive-well-off-rotanev', 'mainnetOwner': '0x487360568fE5Ee4AE5c75782D5F63e9935B0E512', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1692361104, 'startBlock': 9539053, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000003791fa10', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}] Number of sChains on node: 7 Empty sChain structs: 0 ====================================================================================================  [2023-09-07 17:59:36,236 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:36,260 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 17:59:36,290 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 17:59:36,292 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 17:59:36,299 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:59:36,299 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109576.2589266, 'te': 1694109576.2656903, 'initial_status': True} ====================================================================================================  [2023-09-07 17:59:36,301 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:59:36.301405 [2023-09-07 17:59:36,305 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:59:36,306 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:59:36,349 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:59:36,349 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain portly-passionate-sirius - Process is running: PID = 7 [2023-09-07 17:59:36,640 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 17:59:36,726 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:59:36,726 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain curly-red-alterf - Process is running: PID = 8 [2023-09-07 17:59:37,102 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:59:37,102 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain howling-hoedus - Process is running: PID = 9 [2023-09-07 17:59:37,276 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:59:37,277 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:59:37,278 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:59:37.278711 [2023-09-07 17:59:37,286 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 17:59:37,286 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 17:59:37,292 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 17:59:37,296 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109577.285247, 'te': 1694109577.2965913, 'initial_status': True} ====================================================================================================  [2023-09-07 17:59:37,298 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 17:59:37.298674 [2023-09-07 17:59:37,303 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:59:37,303 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:59:37,477 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:59:37,477 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain green-giddy-denebola - Process is running: PID = 10 [2023-09-07 17:59:37,853 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:59:37,853 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain wan-tasty-kitalpha - Process is running: PID = 11 [2023-09-07 17:59:38,225 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:59:38,226 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain free-basic-rosy-SKALE - Process is running: PID = 30622 [2023-09-07 17:59:38,599 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 17:59:38,599 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain submissive-well-off-rotanev - Process is running: PID = 24912 [2023-09-07 17:59:38,600 INFO][1][MainProcess][MainThread] - core.schains.process_manager:88 - Process manager procedure finished [2023-09-07 17:59:38,600 INFO][1][MainProcess][MainThread] - __main__:65 - Sleeping for 90s after run_process_manager [2023-09-07 17:59:44,952 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 17:59:45,118 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 17:59:45,118 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:59:45,118 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 17:59:45,119 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 17:59:45,287 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 17:59:45,288 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 17:59:45,290 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 17:59:45,294 ERROR][11][wan-tasty-kitalpha][T_0] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 17:59:45,295 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:59:45,991 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109552, schain_finish_ts: 1693742304 [2023-09-07 17:59:47,129 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:59:47,130 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:59:47,131 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:59:47,131 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 17:59:47,132 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 17:59:47,132 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 17:59:47,155 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 17:59:47,476 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:47,645 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:47,757 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:59:47,758 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2562 [2023-09-07 17:59:47,758 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 17:59:47,759 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:59:47,759 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:59:47,759 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:59:47,760 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:59:47,768 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:59:47,771 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:59:47.771800 [2023-09-07 17:59:47,946 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:47,981 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 17:59:48,021 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 17:59:48,024 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 17:59:48,031 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:59:48,032 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109587.9793696, 'te': 1694109587.986401, 'initial_status': True} ====================================================================================================  [2023-09-07 17:59:48,034 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 17:59:48.034269 [2023-09-07 17:59:48,038 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:59:48,039 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:59:50,772 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109588, schain_finish_ts: 1693742004 [2023-09-07 17:59:51,436 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109588, schain_finish_ts: 1692377364 [2023-09-07 17:59:51,903 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:59:51,904 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:59:51,904 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:59:51,904 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 17:59:51,905 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 17:59:51,905 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 17:59:51,907 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 17:59:51,911 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 17:59:51,930 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 17:59:52,187 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:52,357 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:52,470 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:59:52,471 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2555 [2023-09-07 17:59:52,472 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 17:59:52,472 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:59:52,472 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:59:52,473 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:59:52,473 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:59:52,479 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:59:52,481 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:59:52.481584 [2023-09-07 17:59:52,552 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 17:59:52,553 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:59:52,553 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 17:59:52,553 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 17:59:52,553 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 17:59:52,554 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 17:59:52,556 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 17:59:52,556 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 17:59:52,592 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 17:59:52,706 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:52,744 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 17:59:52,780 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 17:59:52,782 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 17:59:52,790 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:59:52,791 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109592.7419167, 'te': 1694109592.7499187, 'initial_status': True} ====================================================================================================  [2023-09-07 17:59:52,792 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:59:52.792918 [2023-09-07 17:59:52,797 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:59:52,799 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:59:52,913 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:53,223 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:53,345 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:59:53,346 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2542 [2023-09-07 17:59:53,346 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 17:59:53,347 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 17:59:53,347 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 17:59:53,347 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:59:53,348 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 17:59:53,360 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 17:59:53,364 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:59:53.364775 [2023-09-07 17:59:53,524 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 17:59:53,552 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 17:59:53,592 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 17:59:53,594 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 17:59:53,601 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 17:59:53,602 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109593.5505562, 'te': 1694109593.5613635, 'initial_status': True} ====================================================================================================  [2023-09-07 17:59:53,603 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:59:53.603664 [2023-09-07 17:59:53,608 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 17:59:53,609 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 17:59:53,735 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 17:59:53,735 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:59:53,737 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 17:59:53.737594 [2023-09-07 17:59:53,743 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 17:59:54,415 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 17:59:54,416 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 17:59:54,417 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:59:54.417851 [2023-09-07 17:59:54,424 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 17:59:54,425 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 17:59:54,431 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 17:59:54,436 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109594.4232733, 'te': 1694109594.4360826, 'initial_status': True} ====================================================================================================  [2023-09-07 17:59:54,437 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 17:59:54.437918 [2023-09-07 17:59:54,443 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 17:59:54,444 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 17:59:56,680 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109588, schain_finish_ts: 0 [2023-09-07 17:59:57,830 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 17:59:57,830 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 17:59:57,830 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 18:00:00,386 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109588, schain_finish_ts: 1693412520 [2023-09-07 18:00:01,529 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:00:01,529 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:00:01,529 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:00:01,529 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:00:01,529 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:00:01,530 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:00:01,531 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:00:01,532 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 18:00:01,567 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:00:01,711 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109588, schain_finish_ts: 1692637668 [2023-09-07 18:00:01,784 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:00:02,040 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:00:02,156 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:00:02,157 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2560 [2023-09-07 18:00:02,157 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:00:02,158 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:00:02,158 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:00:02,158 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:00:02,159 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:00:02,167 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:00:02,169 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:00:02.169073 [2023-09-07 18:00:02,357 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:00:02,386 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:00:02,416 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:00:02,418 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:00:02,425 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:00:02,426 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109602.384417, 'te': 1694109602.3937302, 'initial_status': True} ====================================================================================================  [2023-09-07 18:00:02,427 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:00:02.427715 [2023-09-07 18:00:02,432 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:00:02,432 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:00:02,594 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109588, schain_finish_ts: 1694108304 [2023-09-07 18:00:02,860 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:00:02,860 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:00:02,861 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:00:02,861 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:00:02,861 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:00:02,861 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:00:02,868 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:00:02,868 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 18:00:02,904 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:00:03,286 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:00:03,341 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:00:03,341 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:00:03,344 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:00:03.344087 [2023-09-07 18:00:03,355 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:00:03,356 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:00:03,365 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:00:03,370 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109603.352733, 'te': 1694109603.3704424, 'initial_status': True} ====================================================================================================  [2023-09-07 18:00:03,373 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:00:03.373876 [2023-09-07 18:00:03,379 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:00:03,379 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:00:03,509 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:00:03,627 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:00:03,630 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2557 [2023-09-07 18:00:03,631 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:00:03,631 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:00:03,631 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:00:03,632 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:00:03,632 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:00:03,637 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:00:03,639 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:00:03.639388 [2023-09-07 18:00:03,747 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:00:03,747 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:00:03,748 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 18:00:03,787 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:00:03,814 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:00:03,848 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:00:03,851 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:00:03,860 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:00:03,861 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109603.813053, 'te': 1694109603.8198805, 'initial_status': True} ====================================================================================================  [2023-09-07 18:00:03,863 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:00:03.863387 [2023-09-07 18:00:03,868 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:00:03,869 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:00:04,801 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:00:04,801 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:00:04,804 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:00:04.804652 [2023-09-07 18:00:04,813 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:00:04,813 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:00:04,820 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:00:04,826 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109604.811157, 'te': 1694109604.8265083, 'initial_status': True} ====================================================================================================  [2023-09-07 18:00:04,829 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:00:04.829191 [2023-09-07 18:00:04,835 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:00:04,835 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:00:07,587 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent correct data. Waiting for FailedDkg event... [2023-09-07 18:00:10,105 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109600, schain_finish_ts: 1693742304 [2023-09-07 18:00:11,264 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:00:11,264 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:00:11,265 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:00:11,265 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 18:00:11,265 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 18:00:11,265 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:00:11,273 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:00:11,274 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 18:00:11,310 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:00:11,602 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:00:11,837 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:00:11,976 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:00:11,977 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2563 [2023-09-07 18:00:11,978 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:00:11,978 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:00:11,979 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:00:11,979 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:00:11,980 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:00:11,990 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:00:11,993 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:00:11.993927 [2023-09-07 18:00:12,425 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:00:12,457 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:00:12,490 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:00:12,493 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:00:12,503 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:00:12,504 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109612.4558403, 'te': 1694109612.4644585, 'initial_status': True} ====================================================================================================  [2023-09-07 18:00:12,506 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:00:12.506317 [2023-09-07 18:00:12,510 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:00:12,511 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:00:12,857 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 18:00:12,858 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 18:00:13,144 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 18:00:13,144 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:00:13,147 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:00:13.147244 [2023-09-07 18:00:13,153 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:00:20,536 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109600, schain_finish_ts: 1692377364 [2023-09-07 18:00:21,740 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:00:21,740 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:00:21,740 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:00:21,740 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:00:21,741 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:00:21,741 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:00:21,742 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:00:21,743 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 18:00:21,769 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:00:21,981 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:00:22,142 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:00:22,266 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:00:22,267 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2543 [2023-09-07 18:00:22,268 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:00:22,268 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:00:22,268 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:00:22,269 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:00:22,269 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:00:22,282 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:00:22,285 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:00:22.285447 [2023-09-07 18:00:22,465 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:00:22,490 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:00:22,520 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:00:22,522 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:00:22,529 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:00:22,529 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109622.4884412, 'te': 1694109622.4950557, 'initial_status': True} ====================================================================================================  [2023-09-07 18:00:22,531 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:00:22.531329 [2023-09-07 18:00:22,537 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:00:22,538 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:00:23,565 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:00:23,565 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:00:23,568 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:00:23.568301 [2023-09-07 18:00:23,579 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:00:23,579 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:00:23,587 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:00:23,593 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109623.5768564, 'te': 1694109623.5933635, 'initial_status': True} ====================================================================================================  [2023-09-07 18:00:23,597 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:00:23.597909 [2023-09-07 18:00:23,605 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:00:23,605 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:00:24,837 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109600, schain_finish_ts: 1692637668 [2023-09-07 18:00:26,033 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:00:26,033 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:00:26,034 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:00:26,034 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:00:26,034 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:00:26,034 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:00:26,036 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:00:26,039 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 18:00:26,066 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:00:26,371 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:00:26,531 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:00:26,650 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:00:26,651 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2558 [2023-09-07 18:00:26,652 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:00:26,652 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:00:26,653 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:00:26,653 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:00:26,653 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:00:26,660 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:00:26,662 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:00:26.662318 [2023-09-07 18:00:26,875 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:00:26,899 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109624, schain_finish_ts: 1693742004 [2023-09-07 18:00:26,924 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:00:26,967 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:00:26,970 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:00:26,979 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:00:26,980 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109626.9227064, 'te': 1694109626.9303987, 'initial_status': True} ====================================================================================================  [2023-09-07 18:00:26,982 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:00:26.982727 [2023-09-07 18:00:26,987 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:00:26,988 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:00:27,818 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:00:27,818 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:00:27,821 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:00:27.821331 [2023-09-07 18:00:27,831 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:00:27,832 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:00:27,840 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:00:27,845 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109627.8293312, 'te': 1694109627.8454397, 'initial_status': True} ====================================================================================================  [2023-09-07 18:00:27,848 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:00:27.848258 [2023-09-07 18:00:27,854 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:00:27,855 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:00:28,080 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:00:28,080 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:00:28,081 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:00:28,081 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:00:28,081 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:00:28,081 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 18:00:28,103 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:00:28,488 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:00:28,746 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:00:28,878 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:00:28,879 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2556 [2023-09-07 18:00:28,880 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:00:28,880 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:00:28,880 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:00:28,880 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:00:28,881 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:00:28,887 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:00:28,888 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:00:28.888829 [2023-09-07 18:00:29,035 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:00:29,060 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:00:29,091 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:00:29,093 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:00:29,100 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:00:29,100 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109629.0585487, 'te': 1694109629.065946, 'initial_status': True} ====================================================================================================  [2023-09-07 18:00:29,102 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:00:29.102131 [2023-09-07 18:00:29,108 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:00:29,115 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:00:32,853 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 18:00:33,026 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:00:33,026 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 18:00:33,026 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 18:00:33,027 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 18:00:33,154 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 18:00:33,154 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 18:00:33,203 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 18:00:33,204 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 18:00:33,206 INFO][8][curly-red-alterf][T_1] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 18:00:33,210 ERROR][8][curly-red-alterf][T_1] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:00:33,211 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:00:33,482 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109624, schain_finish_ts: 1693412520 [2023-09-07 18:00:34,544 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:40 - Collecting rotation history for submissive-well-off-rotanev... [2023-09-07 18:00:34,616 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:00:34,617 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:00:34,617 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:00:34,617 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:00:34,617 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:00:34,617 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:00:34,619 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:00:34,622 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 18:00:34,653 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:00:34,931 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:00:35,271 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:00:35,374 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:00:35,375 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2561 [2023-09-07 18:00:35,376 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:00:35,376 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:00:35,376 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:00:35,377 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:00:35,377 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:00:35,388 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:00:35,390 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:00:35.390894 [2023-09-07 18:00:35,555 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:00:35,582 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:00:35,623 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:00:35,626 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:00:35,634 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:00:35,635 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109635.5802882, 'te': 1694109635.5866132, 'initial_status': True} ====================================================================================================  [2023-09-07 18:00:35,636 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:00:35.636546 [2023-09-07 18:00:35,640 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:00:35,641 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:00:35,713 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:50 - Rotation data for submissive-well-off-rotanev: Rotation(leaving_node_id=65, new_node_id=65, freeze_until=1694108100, rotation_counter=8) [2023-09-07 18:00:36,433 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:00:36,433 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:00:36,436 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:00:36.436333 [2023-09-07 18:00:36,443 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:00:36,443 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:00:36,449 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:00:36,454 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109636.4419935, 'te': 1694109636.4546158, 'initial_status': True} ====================================================================================================  [2023-09-07 18:00:36,456 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:00:36.456338 [2023-09-07 18:00:36,461 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:00:36,461 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:00:36,714 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109624, schain_finish_ts: 1694108304 [2023-09-07 18:00:36,779 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109624, schain_finish_ts: 0 [2023-09-07 18:00:37,210 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109624, schain_finish_ts: 1693742304 [2023-09-07 18:00:37,851 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:00:37,851 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:00:37,852 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 18:00:37,912 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:00:37,912 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:00:37,912 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 18:00:38,323 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:00:38,323 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:00:38,324 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:00:38,324 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:00:38,324 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:00:38,324 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 18:00:38,376 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:00:38,523 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:197 - sChain free-basic-rosy-SKALE:Complainted node did not send a response.Sending complaint once again [2023-09-07 18:00:38,523 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:326 - sChain: free-basic-rosy-SKALE. 14 node is trying to sent a complaint on 3 node [2023-09-07 18:00:38,588 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:00:38,870 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:00:39,032 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:00:39,039 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2564 [2023-09-07 18:00:39,040 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:00:39,041 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:00:39,041 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:00:39,041 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:00:39,043 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:00:39,049 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:00:39,051 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:00:39.051474 [2023-09-07 18:00:39,326 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:00:39,352 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:00:39,383 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:00:39,385 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:00:39,399 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:00:39,403 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109639.3508136, 'te': 1694109639.3573403, 'initial_status': True} ====================================================================================================  [2023-09-07 18:00:39,411 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:00:39.410995 [2023-09-07 18:00:39,416 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:00:39,417 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:00:39,601 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.transactions.tools:46 - Dry run tx: complaint, sender: 0x523E13D962A7614752eFa3dB62e0466dF0d9E89D, wallet: RedisWalletAdapter, value: 0, [2023-09-07 18:00:40,644 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.transactions.tools:60 - Estimated gas for complaint: 3861543 [2023-09-07 18:00:40,811 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.transactions.tools:126 - Tx: complaint, Fields: {'chainId': 5, 'gas': 3861543, 'nonce': 314, 'value': 0, 'gasPrice': 10000000000, 'type': 1, 'to': '0x69E2C9576B36c8d7938Ba45a1C909B67EC53048E', 'data': '0xd76c2c4f000002bb9043962a56bacebbde03191f32a67a4b888426ac09f83548ae92e2350000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000001f'}, [2023-09-07 18:00:40,812 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.wallets.redis_wallet:135 - Sending {'chainId': 5, 'gas': 3861543, 'nonce': 314, 'value': 0, 'gasPrice': 10000000000, 'type': 1, 'to': '0x69E2C9576B36c8d7938Ba45a1C909B67EC53048E', 'data': '0xd76c2c4f000002bb9043962a56bacebbde03191f32a67a4b888426ac09f83548ae92e2350000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000001f'} to redis pool, method: dkg.complaint [2023-09-07 18:00:40,812 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.wallets.redis_wallet:145 - Adding tx b'tx-d8b5add13422908d' to the pool [2023-09-07 18:00:40,812 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.wallets.redis_wallet:147 - Saving tx b'tx-d8b5add13422908d' record: b'{"status": "PROPOSED", "score": 51694109640, "multiplier": null, "tx_hash": null, "method": "dkg.complaint", "meta": null, "chainId": 5, "gas": null, "nonce": 314, "value": 0, "gasPrice": 10000000000, "type": 1, "to": "0x69E2C9576B36c8d7938Ba45a1C909B67EC53048E", "data": "0xd76c2c4f000002bb9043962a56bacebbde03191f32a67a4b888426ac09f83548ae92e2350000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000001f"}' [2023-09-07 18:00:51,052 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 63 -> 65 [2023-09-07 18:00:53,063 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 18:00:53,249 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:00:53,250 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:00:53,250 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:00:53,251 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 18:00:53,428 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 18:00:53,428 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 18:00:53,433 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 18:00:53,440 ERROR][11][wan-tasty-kitalpha][T_0] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:00:53,441 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:00:56,065 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 23 -> 24 [2023-09-07 18:00:59,034 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109648, schain_finish_ts: 1693742004 [2023-09-07 18:00:59,979 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109648, schain_finish_ts: 1692637668 [2023-09-07 18:01:00,164 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:01:00,164 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:01:00,164 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:01:00,165 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 18:01:00,165 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 18:01:00,165 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:01:00,167 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:01:00,167 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 18:01:00,214 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:01:00,496 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:00,661 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:00,711 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 9 -> 64 [2023-09-07 18:01:00,731 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109648, schain_finish_ts: 1692377364 [2023-09-07 18:01:00,829 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:01:00,830 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2557 [2023-09-07 18:01:00,831 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:01:00,831 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:01:00,832 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:01:00,832 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:01:00,832 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:01:00,840 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:01:00,842 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:01:00.842634 [2023-09-07 18:01:01,097 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:01,137 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:01:01,150 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:01:01,150 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:01:01,150 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:01:01,151 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:01:01,151 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:01:01,151 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:01:01,157 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:01:01,162 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 18:01:01,198 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:01:01,201 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:01:01,205 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:01:01,211 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:01:01,212 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109661.134646, 'te': 1694109661.1513877, 'initial_status': True} ====================================================================================================  [2023-09-07 18:01:01,214 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:01:01.214410 [2023-09-07 18:01:01,221 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:01:01,222 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:01:01,625 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:01,831 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:01,908 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:01:01,908 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:01:01,908 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:01:01,909 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:01:01,909 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:01:01,909 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:01:01,909 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:01:01,910 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 18:01:01,978 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:01:01,985 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 18:01:01,986 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:01:01,989 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:01:01.988998 [2023-09-07 18:01:01,995 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:01:02,037 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:01:02,038 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2559 [2023-09-07 18:01:02,039 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:01:02,040 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:01:02,040 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:01:02,041 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:01:02,042 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:01:02,049 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:01:02,052 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:01:02.052684 [2023-09-07 18:01:02,291 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:02,338 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:01:02,381 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:01:02,385 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:01:02,395 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:01:02,396 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109662.3366501, 'te': 1694109662.3436322, 'initial_status': True} ====================================================================================================  [2023-09-07 18:01:02,398 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:01:02.398386 [2023-09-07 18:01:02,406 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:01:02,407 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:01:02,791 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:02,998 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:03,102 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:01:03,102 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:01:03,105 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:01:03.105725 [2023-09-07 18:01:03,118 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:01:03,118 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:01:03,121 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:01:03,123 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2544 [2023-09-07 18:01:03,124 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:01:03,124 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:01:03,124 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:01:03,125 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:01:03,125 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:01:03,127 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:01:03,132 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109663.1138232, 'te': 1694109663.1320746, 'initial_status': True} ====================================================================================================  [2023-09-07 18:01:03,133 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:01:03,134 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:01:03.134239 [2023-09-07 18:01:03,140 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:01:03,141 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:01:03,145 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:01:03.145502 [2023-09-07 18:01:03,512 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:03,542 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:01:03,576 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:01:03,578 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:01:03,587 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:01:03,588 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109663.5405922, 'te': 1694109663.5503101, 'initial_status': True} ====================================================================================================  [2023-09-07 18:01:03,590 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:01:03.590270 [2023-09-07 18:01:03,596 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:01:03,597 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:01:03,816 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:01:03,816 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:01:03,818 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:01:03.818301 [2023-09-07 18:01:03,826 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:01:03,826 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:01:03,834 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:01:03,841 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109663.8250768, 'te': 1694109663.8410187, 'initial_status': True} ====================================================================================================  [2023-09-07 18:01:03,844 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:01:03.844244 [2023-09-07 18:01:03,852 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:01:03,854 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:01:03,908 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109660, schain_finish_ts: 0 [2023-09-07 18:01:05,113 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:01:05,113 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:01:05,114 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 18:01:05,712 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 35 -> 16 [2023-09-07 18:01:08,693 INFO][1][MainProcess][MainThread] - core.schains.cleaner:65 - Cleaner process started [2023-09-07 18:01:08,740 INFO][30935][cleaner][MainThread] - core.schains.cleaner:110 - Cleaner procedure started. [2023-09-07 18:01:10,604 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109660, schain_finish_ts: 1693412520 [2023-09-07 18:01:11,133 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 55 -> 63 [2023-09-07 18:01:11,763 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:01:11,763 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:01:11,763 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:01:11,763 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:01:11,764 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:01:11,764 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:01:11,770 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:01:11,772 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 18:01:11,805 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:01:12,024 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:12,244 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:12,393 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:01:12,395 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2562 [2023-09-07 18:01:12,396 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:01:12,396 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:01:12,397 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:01:12,398 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:01:12,399 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:01:12,409 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:01:12,413 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:01:12.413529 [2023-09-07 18:01:12,631 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:12,669 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:01:12,716 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:01:12,719 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:01:12,726 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:01:12,727 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109672.666556, 'te': 1694109672.6749103, 'initial_status': True} ====================================================================================================  [2023-09-07 18:01:12,728 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:01:12.728662 [2023-09-07 18:01:12,732 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:01:12,733 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:01:13,578 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:01:13,579 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:01:13,581 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:01:13.581614 [2023-09-07 18:01:13,590 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:01:13,590 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:01:13,596 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:01:13,600 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109673.5879562, 'te': 1694109673.600321, 'initial_status': True} ====================================================================================================  [2023-09-07 18:01:13,602 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:01:13.602561 [2023-09-07 18:01:13,606 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:01:13,607 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:01:14,490 INFO][30935][cleaner][MainThread] - core.schains.cleaner:116 - sChains on contracts: ['portly-passionate-sirius', 'curly-red-alterf', 'howling-hoedus', 'green-giddy-denebola', 'wan-tasty-kitalpha', 'free-basic-rosy-SKALE', 'submissive-well-off-rotanev'] sChains on node: ['curly-red-alterf', 'free-basic-rosy-SKALE', 'green-giddy-denebola', 'howling-hoedus', 'portly-passionate-sirius', 'submissive-well-off-rotanev', 'wan-tasty-kitalpha'] [2023-09-07 18:01:14,491 INFO][30935][cleaner][MainThread] - core.schains.cleaner:132 - Cleanup procedure finished [2023-09-07 18:01:14,504 INFO][1][MainProcess][MainThread] - core.schains.cleaner:67 - Cleaner process is joined. [2023-09-07 18:01:14,504 INFO][1][MainProcess][MainThread] - core.schains.cleaner:68 - Terminating the process [2023-09-07 18:01:14,505 INFO][1][MainProcess][MainThread] - __main__:70 - Sleeping for 90s after run_cleaner [2023-09-07 18:01:16,899 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 14 -> 11 [2023-09-07 18:01:17,276 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109672, schain_finish_ts: 1693742304 [2023-09-07 18:01:17,844 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109672, schain_finish_ts: 1694108304 [2023-09-07 18:01:18,422 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:01:18,423 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:01:18,423 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:01:18,423 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 18:01:18,423 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 18:01:18,424 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:01:18,426 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:01:18,428 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 18:01:18,451 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:01:18,653 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:18,870 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:19,008 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:01:19,009 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2565 [2023-09-07 18:01:19,009 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:01:19,010 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:01:19,010 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:01:19,010 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:01:19,011 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:01:19,017 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:01:19,019 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:01:19.019868 [2023-09-07 18:01:19,043 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:01:19,044 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:01:19,045 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 18:01:19,324 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:19,359 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:01:19,395 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:01:19,397 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:01:19,407 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:01:19,407 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109679.3579113, 'te': 1694109679.3657053, 'initial_status': True} ====================================================================================================  [2023-09-07 18:01:19,409 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:01:19.409076 [2023-09-07 18:01:19,415 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:01:19,422 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:01:20,216 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 18:01:20,217 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:01:20,219 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:01:20.219010 [2023-09-07 18:01:20,223 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:01:20,863 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 18:01:20,863 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 18:01:22,139 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109672, schain_finish_ts: 1692637668 [2023-09-07 18:01:23,211 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 56 -> 60 [2023-09-07 18:01:23,304 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:01:23,305 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:01:23,305 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:01:23,305 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:01:23,305 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:01:23,306 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:01:23,307 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:01:23,310 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 18:01:23,333 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:01:23,570 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:23,787 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:23,945 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:01:23,946 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2560 [2023-09-07 18:01:23,947 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:01:23,948 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:01:23,948 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:01:23,948 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:01:23,949 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:01:23,957 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:01:23,961 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:01:23.961792 [2023-09-07 18:01:24,327 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:24,357 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:01:24,409 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:01:24,411 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:01:24,419 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:01:24,420 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109684.3553145, 'te': 1694109684.3633425, 'initial_status': True} ====================================================================================================  [2023-09-07 18:01:24,421 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:01:24.421690 [2023-09-07 18:01:24,426 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:01:24,433 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:01:25,139 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:01:25,139 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:01:25,142 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:01:25.142729 [2023-09-07 18:01:25,153 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:01:25,153 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:01:25,161 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:01:25,171 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109685.1506233, 'te': 1694109685.1714559, 'initial_status': True} ====================================================================================================  [2023-09-07 18:01:25,174 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:01:25.174960 [2023-09-07 18:01:25,182 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:01:25,183 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:01:29,402 ERROR][24912][submissive-well-off-rotanev][T_0] - core.schains.task:26 - Task submissive-well-off-rotanev-config failed with max() arg is an empty sequence Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 63, in execute self.am.upstream_config() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 194, in upstream_config new_config = create_new_upstream_config( File "/usr/src/admin/core/schains/config/main.py", line 51, in create_new_upstream_config schain_config = generate_schain_config_with_skale( File "/usr/src/admin/core/schains/config/generator.py", line 213, in generate_schain_config_with_skale node_groups = get_previous_schain_groups(skale, schain_name) File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 56, in get_previous_schain_groups _add_previous_schain_rotations_state( File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 119, in _add_previous_schain_rotations_state new_node_id = max(previous_nodes.items(), key=lambda x: x[1]['finish_ts'])[0] ValueError: max() arg is an empty sequence [2023-09-07 18:01:29,403 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:01:29,912 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109684, schain_finish_ts: 1692377364 [2023-09-07 18:01:31,066 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:01:31,067 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:01:31,067 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:01:31,067 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:01:31,067 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:01:31,068 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:01:31,075 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:01:31,076 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 18:01:31,108 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:01:31,397 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:31,638 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:31,825 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:01:31,828 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2545 [2023-09-07 18:01:31,835 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:01:31,836 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:01:31,836 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:01:31,836 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:01:31,837 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:01:31,845 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:01:31,847 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:01:31.847583 [2023-09-07 18:01:32,026 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:32,055 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:01:32,088 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:01:32,091 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:01:32,099 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:01:32,100 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109692.0541244, 'te': 1694109692.0623486, 'initial_status': True} ====================================================================================================  [2023-09-07 18:01:32,110 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:01:32.110573 [2023-09-07 18:01:32,118 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:01:32,118 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:01:32,549 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109684, schain_finish_ts: 1693742004 [2023-09-07 18:01:33,029 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:01:33,029 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:01:33,033 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:01:33.033606 [2023-09-07 18:01:33,044 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:01:33,044 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:01:33,052 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:01:33,057 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109693.0416565, 'te': 1694109693.0572655, 'initial_status': True} ====================================================================================================  [2023-09-07 18:01:33,059 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:01:33.059781 [2023-09-07 18:01:33,065 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:01:33,065 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:01:33,850 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:01:33,850 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:01:33,851 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:01:33,851 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:01:33,851 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:01:33,852 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 18:01:33,881 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:01:34,342 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:34,533 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:34,675 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:01:34,676 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2558 [2023-09-07 18:01:34,677 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:01:34,677 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:01:34,677 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:01:34,677 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:01:34,678 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:01:34,685 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:01:34,688 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:01:34.688650 [2023-09-07 18:01:34,849 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:34,856 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109684, schain_finish_ts: 1693412520 [2023-09-07 18:01:34,894 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:01:34,928 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:01:34,930 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:01:34,946 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:01:34,947 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109694.881983, 'te': 1694109694.9001079, 'initial_status': True} ====================================================================================================  [2023-09-07 18:01:34,955 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:01:34.955661 [2023-09-07 18:01:34,960 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:01:34,960 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:01:36,005 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:01:36,006 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:01:36,006 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:01:36,006 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:01:36,006 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:01:36,007 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:01:36,010 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:01:36,013 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 18:01:36,040 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:01:36,397 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:36,547 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:36,744 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:01:36,745 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2563 [2023-09-07 18:01:36,745 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:01:36,746 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:01:36,746 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:01:36,746 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:01:36,747 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:01:36,754 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:01:36,756 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:01:36.756194 [2023-09-07 18:01:36,937 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:36,963 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:01:37,005 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:01:37,009 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:01:37,019 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:01:37,019 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109696.9616525, 'te': 1694109696.9687345, 'initial_status': True} ====================================================================================================  [2023-09-07 18:01:37,022 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:01:37.022445 [2023-09-07 18:01:37,028 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:01:37,029 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:01:37,914 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:01:37,914 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:01:37,917 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:01:37.917022 [2023-09-07 18:01:37,924 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:01:37,924 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:01:37,931 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:01:37,936 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109697.9227602, 'te': 1694109697.9363778, 'initial_status': True} ====================================================================================================  [2023-09-07 18:01:37,938 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:01:37.938643 [2023-09-07 18:01:37,943 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:01:37,943 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:01:39,348 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 18:01:39,348 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 18:01:40,908 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 18:01:41,077 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109696, schain_finish_ts: 0 [2023-09-07 18:01:41,088 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:01:41,088 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 18:01:41,088 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 18:01:41,088 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 18:01:41,256 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 18:01:41,257 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 18:01:41,260 INFO][8][curly-red-alterf][T_0] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 18:01:41,266 ERROR][8][curly-red-alterf][T_0] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:01:41,270 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:01:42,232 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:01:42,233 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:01:42,233 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:40 - Task submissive-well-off-rotanev-config finished with None [2023-09-07 18:01:42,233 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:42 - Running task submissive-well-off-rotanev-config [2023-09-07 18:01:42,233 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 18:01:43,983 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': False} [2023-09-07 18:01:43,983 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:01:43,986 INFO][24912][submissive-well-off-rotanev][T_0] - web.models.schain:152 - Changing monitor_last_seen for submissive-well-off-rotanev to 2023-09-07 18:01:43.986671 [2023-09-07 18:01:43,996 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.action:190 - Creating new upstream_config rotation_id: 8, stream: regression-2.2 [2023-09-07 18:01:43,999 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.config.main:49 - Generating sChain config for submissive-well-off-rotanev [2023-09-07 18:01:49,351 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109696, schain_finish_ts: 1693742304 [2023-09-07 18:01:50,516 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:01:50,516 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:01:50,517 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:01:50,517 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:01:50,517 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:01:50,517 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 18:01:50,541 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:01:50,749 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:50,926 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:51,051 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:01:51,052 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2566 [2023-09-07 18:01:51,053 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:01:51,053 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:01:51,053 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:01:51,053 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:01:51,054 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:01:51,063 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:01:51,066 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:01:51.066038 [2023-09-07 18:01:51,263 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:51,290 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:01:51,321 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:01:51,323 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:01:51,332 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:01:51,332 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109711.2887638, 'te': 1694109711.2968671, 'initial_status': True} ====================================================================================================  [2023-09-07 18:01:51,334 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:01:51.334175 [2023-09-07 18:01:51,338 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:01:51,339 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:01:53,085 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109696, schain_finish_ts: 1694108304 [2023-09-07 18:01:54,301 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:01:54,302 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:01:54,304 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 18:01:55,799 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109696, schain_finish_ts: 1693742004 [2023-09-07 18:01:56,968 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:01:56,969 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:01:56,969 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:01:56,969 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 18:01:56,969 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 18:01:56,970 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:01:56,971 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:01:56,973 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 18:01:57,005 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:01:57,253 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:57,467 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:57,593 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:01:57,596 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2559 [2023-09-07 18:01:57,597 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:01:57,597 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:01:57,598 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:01:57,599 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:01:57,600 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:01:57,616 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:01:57,621 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:01:57.621242 [2023-09-07 18:01:57,831 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:01:57,864 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:01:57,906 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:01:57,909 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:01:57,920 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:01:57,920 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109717.8623142, 'te': 1694109717.8701875, 'initial_status': True} ====================================================================================================  [2023-09-07 18:01:57,923 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:01:57.923110 [2023-09-07 18:01:57,927 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:01:57,928 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:01:58,752 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 18:01:58,752 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:01:58,755 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:01:58.755379 [2023-09-07 18:01:58,760 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:01:59,052 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 18:01:59,237 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:01:59,237 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:01:59,238 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:01:59,238 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 18:01:59,288 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109696, schain_finish_ts: 1692637668 [2023-09-07 18:01:59,415 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 18:01:59,416 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 18:01:59,428 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 18:01:59,438 ERROR][11][wan-tasty-kitalpha][T_1] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:01:59,439 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:02:00,456 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:02:00,457 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:02:00,457 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:02:00,457 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:02:00,457 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:02:00,458 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:02:00,459 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:02:00,460 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 18:02:00,494 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:02:00,732 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:01,055 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:01,179 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:02:01,180 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2561 [2023-09-07 18:02:01,181 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:02:01,181 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:02:01,181 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:02:01,182 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:02:01,182 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:02:01,188 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:02:01,190 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:02:01.190335 [2023-09-07 18:02:01,370 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:01,396 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:02:01,432 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:02:01,441 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:02:01,449 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:02:01,450 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109721.3943388, 'te': 1694109721.401274, 'initial_status': True} ====================================================================================================  [2023-09-07 18:02:01,451 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:02:01.451949 [2023-09-07 18:02:01,456 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:02:01,457 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:02:02,256 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:02:02,257 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:02:02,258 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:02:02.258910 [2023-09-07 18:02:02,265 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:02:02,265 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:02:02,271 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:02:02,275 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109722.263959, 'te': 1694109722.275381, 'initial_status': True} ====================================================================================================  [2023-09-07 18:02:02,277 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:02:02.277727 [2023-09-07 18:02:02,282 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:02:02,282 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:02:10,066 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109720, schain_finish_ts: 1692377364 [2023-09-07 18:02:11,237 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:02:11,237 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:02:11,237 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:02:11,237 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:02:11,238 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:02:11,238 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:02:11,241 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:02:11,243 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 18:02:11,289 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:02:11,824 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:12,068 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:12,207 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:02:12,208 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2546 [2023-09-07 18:02:12,209 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:02:12,209 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:02:12,210 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:02:12,210 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:02:12,211 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:02:12,224 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:02:12,227 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:02:12.227440 [2023-09-07 18:02:12,516 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:12,546 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:02:12,586 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:02:12,589 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:02:12,599 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:02:12,600 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109732.5448103, 'te': 1694109732.5527632, 'initial_status': True} ====================================================================================================  [2023-09-07 18:02:12,602 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:02:12.602617 [2023-09-07 18:02:12,611 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:02:12,612 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:02:13,008 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109720, schain_finish_ts: 1693412520 [2023-09-07 18:02:13,133 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:02:13,133 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:02:13,136 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:02:13.136620 [2023-09-07 18:02:13,146 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:02:13,146 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:02:13,153 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:02:13,158 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109733.1436996, 'te': 1694109733.1580715, 'initial_status': True} ====================================================================================================  [2023-09-07 18:02:13,160 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:02:13.160053 [2023-09-07 18:02:13,165 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:02:13,165 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:02:14,153 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:02:14,153 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:02:14,153 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:02:14,154 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:02:14,154 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:02:14,154 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:02:14,175 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:02:14,179 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 18:02:14,210 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:02:14,490 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:14,649 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:14,767 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:02:14,768 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2564 [2023-09-07 18:02:14,769 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:02:14,769 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:02:14,769 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:02:14,770 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:02:14,770 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:02:14,777 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:02:14,779 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:02:14.779756 [2023-09-07 18:02:15,021 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:15,052 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:02:15,200 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109732, schain_finish_ts: 0 [2023-09-07 18:02:15,237 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.transactions.tools:190 - Tx complaint completed after 1/3 retries [2023-09-07 18:02:15,237 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:354 - sChain: free-basic-rosy-SKALE. 14 node sent a complaint on 3 node [2023-09-07 18:02:15,237 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:259 - sChain: free-basic-rosy-SKALE. Will wait for FailedDkg event [2023-09-07 18:02:15,239 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:02:15,241 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:02:15,251 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:02:15,252 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109735.0513563, 'te': 1694109735.0604506, 'initial_status': True} ====================================================================================================  [2023-09-07 18:02:15,254 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:02:15.254337 [2023-09-07 18:02:15,260 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:02:15,261 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:02:15,858 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:265 - sChain: free-basic-rosy-SKALE. Not all nodes sent response. Waiting for FailedDkg event... [2023-09-07 18:02:15,993 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:02:15,993 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:02:15,995 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:02:15.995469 [2023-09-07 18:02:16,004 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:02:16,004 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:02:16,011 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:02:16,017 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109736.0024002, 'te': 1694109736.0168622, 'initial_status': True} ====================================================================================================  [2023-09-07 18:02:16,019 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:02:16.019751 [2023-09-07 18:02:16,030 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:02:16,030 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:02:16,332 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:02:16,333 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:02:16,333 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 18:02:16,623 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.main:171 - sChain free-basic-rosy-SKALE DKG procedure failed with sChain: free-basic-rosy-SKALE. Dkg failed due to event FailedDKG [2023-09-07 18:02:16,626 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:118 - Changing DKG status for free-basic-rosy-SKALE to FAILED [2023-09-07 18:02:16,632 ERROR][30622][free-basic-rosy-SKALE][T_0] - core.schains.task:26 - Task free-basic-rosy-SKALE-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:02:16,633 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:02:17,942 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 18:02:17,943 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 18:02:20,489 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109732, schain_finish_ts: 1693742304 [2023-09-07 18:02:21,628 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:02:21,628 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:02:21,628 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:02:21,628 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 18:02:21,629 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 18:02:21,629 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:02:21,630 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:02:21,631 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 18:02:21,666 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:02:21,901 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:22,268 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:22,408 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:02:22,412 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2567 [2023-09-07 18:02:22,413 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:02:22,413 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:02:22,414 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:02:22,414 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:02:22,414 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:02:22,420 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:02:22,422 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:02:22.422437 [2023-09-07 18:02:22,657 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:22,701 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:02:22,744 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:02:22,747 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:02:22,760 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:02:22,761 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109742.6997597, 'te': 1694109742.7062504, 'initial_status': True} ====================================================================================================  [2023-09-07 18:02:22,766 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:02:22.766103 [2023-09-07 18:02:22,772 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:02:22,773 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:02:23,557 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 18:02:23,557 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:02:23,560 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:02:23.560739 [2023-09-07 18:02:23,565 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:02:27,479 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109744, schain_finish_ts: 1692637668 [2023-09-07 18:02:28,605 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:02:28,605 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:02:28,605 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:02:28,605 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:02:28,605 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:02:28,605 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:02:28,610 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:02:28,614 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 18:02:28,647 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:02:28,895 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:29,082 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:29,206 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:02:29,207 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2562 [2023-09-07 18:02:29,208 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:02:29,208 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:02:29,208 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:02:29,209 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:02:29,209 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:02:29,218 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:02:29,220 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:02:29.220271 [2023-09-07 18:02:29,420 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109744, schain_finish_ts: 1694108304 [2023-09-07 18:02:29,518 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:29,545 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:02:29,576 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:02:29,579 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:02:29,607 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:02:29,611 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109749.5438604, 'te': 1694109749.5507956, 'initial_status': True} ====================================================================================================  [2023-09-07 18:02:29,616 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:02:29.616960 [2023-09-07 18:02:29,622 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:02:29,622 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:02:30,049 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109744, schain_finish_ts: 1693742004 [2023-09-07 18:02:30,421 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:02:30,421 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:02:30,423 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:02:30.423503 [2023-09-07 18:02:30,432 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:02:30,432 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:02:30,438 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:02:30,444 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109750.4305818, 'te': 1694109750.4439113, 'initial_status': True} ====================================================================================================  [2023-09-07 18:02:30,446 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:02:30.446183 [2023-09-07 18:02:30,453 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:02:30,453 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:02:30,597 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:02:30,598 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:02:30,598 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.task:40 - Task free-basic-rosy-SKALE-config finished with None [2023-09-07 18:02:30,598 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.task:42 - Running task free-basic-rosy-SKALE-config [2023-09-07 18:02:30,598 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 18:02:31,161 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:02:31,161 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:02:31,161 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:02:31,164 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:02:31,165 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:02:31,166 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 18:02:31,192 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:02:31,472 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:31,641 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:31,748 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:02:31,750 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2560 [2023-09-07 18:02:31,751 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:02:31,751 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:02:31,752 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:02:31,752 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:02:31,752 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:02:31,762 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:02:31,765 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:02:31.765266 [2023-09-07 18:02:31,975 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:32,005 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:02:32,035 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:02:32,037 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:02:32,043 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:02:32,044 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109752.0035243, 'te': 1694109752.0110736, 'initial_status': True} ====================================================================================================  [2023-09-07 18:02:32,045 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:02:32.045800 [2023-09-07 18:02:32,050 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:02:32,050 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:02:32,338 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': False, 'upstream_config': False} [2023-09-07 18:02:32,338 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:02:32,340 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:152 - Changing monitor_last_seen for free-basic-rosy-SKALE to 2023-09-07 18:02:32.340822 [2023-09-07 18:02:32,345 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:02:37,397 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 18:02:37,580 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:02:37,580 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 18:02:37,580 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 18:02:37,581 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 18:02:37,756 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 18:02:37,757 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 18:02:37,763 INFO][8][curly-red-alterf][T_1] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 18:02:37,788 ERROR][8][curly-red-alterf][T_1] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:02:37,789 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:02:40,235 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109756, schain_finish_ts: 1692377364 [2023-09-07 18:02:41,753 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:02:41,753 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:02:41,754 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:02:41,754 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:02:41,754 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:02:41,754 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:02:41,754 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:02:41,756 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 18:02:41,787 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:02:41,991 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:42,154 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:42,238 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109756, schain_finish_ts: 1693412520 [2023-09-07 18:02:42,273 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:02:42,274 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2547 [2023-09-07 18:02:42,275 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:02:42,275 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:02:42,275 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:02:42,275 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:02:42,276 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:02:42,282 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:02:42,284 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:02:42.284653 [2023-09-07 18:02:42,451 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:42,488 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:02:42,527 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:02:42,530 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:02:42,547 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:02:42,548 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109762.4869015, 'te': 1694109762.4937484, 'initial_status': True} ====================================================================================================  [2023-09-07 18:02:42,550 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:02:42.550899 [2023-09-07 18:02:42,568 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:02:42,570 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:02:43,510 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 18:02:43,510 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 18:02:43,631 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:02:43,631 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:02:43,631 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:02:43,632 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:02:43,632 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:02:43,632 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:02:43,636 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:02:43,639 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 18:02:43,680 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:02:43,816 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:02:43,816 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:02:43,818 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:02:43.818842 [2023-09-07 18:02:43,828 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:02:43,828 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:02:43,840 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:02:43,846 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109763.8265784, 'te': 1694109763.846472, 'initial_status': True} ====================================================================================================  [2023-09-07 18:02:43,848 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:02:43.848486 [2023-09-07 18:02:43,854 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:02:43,854 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:02:43,980 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:44,149 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:44,273 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:02:44,274 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2565 [2023-09-07 18:02:44,275 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:02:44,275 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:02:44,275 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:02:44,276 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:02:44,276 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:02:44,282 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:02:44,285 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:02:44.285546 [2023-09-07 18:02:44,456 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:44,484 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:02:44,523 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:02:44,525 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:02:44,532 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:02:44,533 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109764.4833095, 'te': 1694109764.4899561, 'initial_status': True} ====================================================================================================  [2023-09-07 18:02:44,534 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:02:44.534950 [2023-09-07 18:02:44,539 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:02:44,540 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:02:44,590 INFO][1][MainProcess][MainThread] - core.schains.process_manager:57 - Process manager started [2023-09-07 18:02:44,804 INFO][1][MainProcess][MainThread] - core.schains.notifications:35 - Node account has 407166861192858363 WEI [2023-09-07 18:02:44,806 INFO][1][MainProcess][MainThread] - tools.notifications.messages:188 - Saving new balance state 3019 1 [2023-09-07 18:02:44,806 INFO][1][MainProcess][MainThread] - core.schains.process_manager:95 - Fetching schains to monitor... [2023-09-07 18:02:45,421 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:02:45,422 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:02:45,424 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:02:45.424555 [2023-09-07 18:02:45,432 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:02:45,432 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:02:45,439 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:02:45,445 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109765.4306371, 'te': 1694109765.4451504, 'initial_status': True} ====================================================================================================  [2023-09-07 18:02:45,447 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:02:45.447444 [2023-09-07 18:02:45,452 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:02:45,452 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:02:46,325 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109756, schain_finish_ts: 0 [2023-09-07 18:02:47,211 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109756, schain_finish_ts: 1693742304 [2023-09-07 18:02:47,468 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:02:47,468 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:02:47,468 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 18:02:49,710 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:02:49,710 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:02:49,710 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:02:49,711 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:02:49,711 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:02:49,711 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 18:02:49,734 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:02:49,935 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:50,071 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:50,188 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:02:50,190 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2568 [2023-09-07 18:02:50,191 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:02:50,192 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:02:50,192 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:02:50,193 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:02:50,193 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:02:50,201 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:02:50,204 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:02:50.204909 [2023-09-07 18:02:50,358 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:02:50,400 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:02:50,445 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:02:50,447 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:02:50,455 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:02:50,456 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109770.39537, 'te': 1694109770.407221, 'initial_status': True} ====================================================================================================  [2023-09-07 18:02:50,460 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:02:50.460303 [2023-09-07 18:02:50,467 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:02:50,468 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:02:50,488 INFO][1][MainProcess][MainThread] - core.schains.process_manager:109 - Get leaving_history for node ... [2023-09-07 18:02:51,515 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:159 - sChain: free-basic-rosy-SKALE. Node id on chain is 14; Node id on contract is 16 [2023-09-07 18:02:51,515 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:162 - sChain: free-basic-rosy-SKALE. DKG timeout is 1800 [2023-09-07 18:02:52,475 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109768, schain_finish_ts: 1692377616 [2023-09-07 18:02:54,452 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109768, schain_finish_ts: 0 [2023-09-07 18:02:54,831 INFO][1][MainProcess][MainThread] - core.schains.process_manager:117 - Got leaving sChains for the node: [] [2023-09-07 18:02:54,832 INFO][1][MainProcess][MainThread] - core.schains.process_manager:101 -  ==================================================================================================== Monitoring sChains Node ID: 16 sChains on node: [{'name': 'portly-passionate-sirius', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666281384, 'startBlock': 7803380, 'deposit': 0, 'index': 1, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x0000017723ca2', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'curly-red-alterf', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1676058936, 'startBlock': 8469646, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000001fe6cdfa', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'howling-hoedus', 'mainnetOwner': '0x1026467C91fdA40aCc7e6935d1E52f67274FD669', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666181400, 'startBlock': 7796660, 'deposit': 0, 'index': 0, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0xe5a600f4c2608', 'multitransactionMode': False, 'thresholdEncryption': False, 'active': True}, {'name': 'green-giddy-denebola', 'mainnetOwner': '0x8e835d5d9E28ffD259E8Fa8988Cacfc333A582B4', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674565164, 'startBlock': 8368049, 'deposit': 0, 'index': 4, 'generation': 1, 'originator': '0xd2001DAb6898127Be2F167B548691C87251D13C3', 'chainId': '0x00000585eb4b1', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'wan-tasty-kitalpha', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674568392, 'startBlock': 8368271, 'deposit': 0, 'index': 5, 'generation': 1, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x00000152e7be9', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'free-basic-rosy-SKALE', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1693996920, 'startBlock': 9646443, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000002bb90439', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'submissive-well-off-rotanev', 'mainnetOwner': '0x487360568fE5Ee4AE5c75782D5F63e9935B0E512', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1692361104, 'startBlock': 9539053, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000003791fa10', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}] Number of sChains on node: 7 Empty sChain structs: 0 ====================================================================================================  [2023-09-07 18:02:55,204 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:02:55,204 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain portly-passionate-sirius - Process is running: PID = 7 [2023-09-07 18:02:55,576 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:02:55,576 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain curly-red-alterf - Process is running: PID = 8 [2023-09-07 18:02:55,947 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:02:55,947 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain howling-hoedus - Process is running: PID = 9 [2023-09-07 18:02:56,323 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:02:56,324 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain green-giddy-denebola - Process is running: PID = 10 [2023-09-07 18:02:56,695 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:02:56,696 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain wan-tasty-kitalpha - Process is running: PID = 11 [2023-09-07 18:02:57,067 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:02:57,067 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain free-basic-rosy-SKALE - Process is running: PID = 30622 [2023-09-07 18:02:57,449 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:02:57,449 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain submissive-well-off-rotanev - Process is running: PID = 24912 [2023-09-07 18:02:57,450 INFO][1][MainProcess][MainThread] - core.schains.process_manager:88 - Process manager procedure finished [2023-09-07 18:02:57,450 INFO][1][MainProcess][MainThread] - __main__:65 - Sleeping for 90s after run_process_manager [2023-09-07 18:03:01,566 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109768, schain_finish_ts: 1692637668 [2023-09-07 18:03:01,570 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109768, schain_finish_ts: 1694108304 [2023-09-07 18:03:02,725 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:03:02,726 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:03:02,726 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 18:03:02,743 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:03:02,743 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:03:02,744 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:03:02,744 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:03:02,744 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:03:02,745 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:03:02,748 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:03:02,748 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 18:03:02,790 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:03:03,041 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:03,223 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:03,357 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:03:03,358 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2563 [2023-09-07 18:03:03,359 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:03:03,359 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:03:03,360 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:03:03,360 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:03:03,360 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:03:03,367 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:03:03,370 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:03:03.370248 [2023-09-07 18:03:03,550 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:03,583 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:03:03,655 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:03:03,657 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:03:03,668 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:03:03,669 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109783.5815153, 'te': 1694109783.5904977, 'initial_status': True} ====================================================================================================  [2023-09-07 18:03:03,671 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:03:03.671351 [2023-09-07 18:03:03,677 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:03:03,680 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:03:04,170 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109768, schain_finish_ts: 1693742004 [2023-09-07 18:03:04,509 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:03:04,509 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:03:04,511 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:03:04.511582 [2023-09-07 18:03:04,518 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:03:04,519 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:03:04,526 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:03:04,531 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109784.5173707, 'te': 1694109784.5309827, 'initial_status': True} ====================================================================================================  [2023-09-07 18:03:04,533 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:03:04.533124 [2023-09-07 18:03:04,538 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:03:04,538 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:03:04,790 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 18:03:04,964 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:03:04,964 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:03:04,965 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:03:04,965 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 18:03:05,152 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 18:03:05,152 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 18:03:05,165 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 18:03:05,174 ERROR][11][wan-tasty-kitalpha][T_0] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:03:05,175 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:03:05,317 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:03:05,317 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:03:05,317 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:03:05,317 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 18:03:05,318 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 18:03:05,318 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:03:05,319 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:03:05,321 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 18:03:05,346 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:03:05,571 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:05,745 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:05,852 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:03:05,853 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2561 [2023-09-07 18:03:05,854 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:03:05,854 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:03:05,854 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:03:05,855 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:03:05,855 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:03:05,862 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:03:05,864 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:03:05.864500 [2023-09-07 18:03:06,254 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:06,310 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:03:06,356 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:03:06,358 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:03:06,368 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:03:06,368 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109786.3076034, 'te': 1694109786.3161218, 'initial_status': True} ====================================================================================================  [2023-09-07 18:03:06,370 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:03:06.370459 [2023-09-07 18:03:06,375 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:03:06,376 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:03:06,708 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109768, schain_finish_ts: 1692377364 [2023-09-07 18:03:07,181 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 18:03:07,182 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:03:07,184 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:03:07.184850 [2023-09-07 18:03:07,189 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:03:07,849 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:03:07,849 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:03:07,849 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:03:07,850 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:03:07,850 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:03:07,850 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:03:07,852 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:03:07,855 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 18:03:07,878 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:03:08,195 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:08,366 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:08,484 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:03:08,485 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2548 [2023-09-07 18:03:08,486 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:03:08,486 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:03:08,486 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:03:08,486 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:03:08,487 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:03:08,493 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:03:08,495 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:03:08.495508 [2023-09-07 18:03:08,648 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:08,683 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:03:08,710 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:03:08,712 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:03:08,719 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:03:08,720 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109788.6812193, 'te': 1694109788.6876485, 'initial_status': True} ====================================================================================================  [2023-09-07 18:03:08,721 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:03:08.721489 [2023-09-07 18:03:08,727 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:03:08,729 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:03:09,426 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109768, schain_finish_ts: 0 [2023-09-07 18:03:09,641 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:03:09,642 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:03:09,644 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:03:09.643978 [2023-09-07 18:03:09,651 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:03:09,651 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:03:09,658 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:03:09,662 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109789.650191, 'te': 1694109789.6625276, 'initial_status': True} ====================================================================================================  [2023-09-07 18:03:09,664 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:03:09.664602 [2023-09-07 18:03:09,670 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:03:09,670 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:03:10,597 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:03:10,597 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:03:10,598 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 18:03:11,154 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.main:162 - Starting dkg procedure for free-basic-rosy-SKALE [2023-09-07 18:03:11,952 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:77 - sChain free-basic-rosy-SKALE: Sending broadcast [2023-09-07 18:03:12,340 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'isPolyExists', 'params': {'polyName': 'POLY:SCHAIN_ID:18860195493846376099765944407454631582519828060563062387235736098366005:NODE_ID:14:DKG_ID:3'}} [2023-09-07 18:03:12,519 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'IsExist': False, 'errorMessage': '', 'status': 0}} [2023-09-07 18:03:12,519 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'generateDKGPoly', 'params': {'polyName': 'POLY:SCHAIN_ID:18860195493846376099765944407454631582519828060563062387235736098366005:NODE_ID:14:DKG_ID:3', 't': 11}} [2023-09-07 18:03:12,701 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'status': 0}} [2023-09-07 18:03:13,455 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getVerificationVector', 'params': {'polyName': 'POLY:SCHAIN_ID:18860195493846376099765944407454631582519828060563062387235736098366005:NODE_ID:14:DKG_ID:3', 'n': 16, 't': 11}} [2023-09-07 18:03:13,666 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'status': 0, 'verificationVector': [['8218553367624260300833602443676579770886206344867283495280064311693369555487', '264355336810953195036999779363856236736704894222055910663320304714991304013', '17295288347106134030507079836446159923302692604746642999546830260693552473207', '15548267806892002078677162908853315555921751482017560009180931830287434640088'], ['15583832325384677179780725693969377274033071491191684490259692889901270409330', '7988063389089544200743250486877982196643752301517667100387181403106675376415', '654644844810683270944738764906617700052694770679142341745642497373245160117', '18932241259947640143829536856637645916999320639070051707404474891687001650547'], ['1452327071259158997347994868596057857053566457360819058170821550460889589348', '3682231151118841798381842353101531849037792164127370438940431433898479495442', '20098573224273324091381175698648727611324877742541905010967497936033526193620', '21650113567391403979467330785971029591626888088039469460679125712222856335279'], ['17796209251087701415891765689277846918297502354316532823071633643120492149348', '21739833469497143012626485001266306326482380154751356749235787136125224351633', '13113995093342755875357217583631577506830604483135569047650343152263002466910', '4102757193806006175071516291052132977742039811755838101525695812647403297188'], ['8225990497162206860817621771871991209565114427618615977527148605888170876999', '18190371008991692017459752915312562304990740821184466203912764764496866097087', '462680252862143297745507070428015128643819213444820765713933639002468336572', '68599317022171312870514176120094835274654365516155951893381084072513601849'], ['14469033027171190466879170689641169238400103280358787152873188728074767730834', '4203735582531337453151930455008123595998068044868809027992137990969462937279', '17996726191053626241987059260952358949457380436094484047103159594058155882728', '17542705191757086177568684219262516485625390154816981609118967905440077931179'], ['4651997535380383285752631970496800560558846999284350964890226665821416355634', '16776122133814776352074539710727413876333908127480166981558482733714833449509', '8149595057460009054245375550988581964359182015394412051174567948332920190606', '17749128551351768086913008167219103561152816234087843604237812066926649901600'], ['21253720251274835977202907533707501051335069340190518821565028642222013829476', '11740889874390806684111875043936088503388732570846501205570233847498382767571', '11684158196186885938286025272514082847384025801198162470202729825769486721488', '8096095313728876162096450046744565788792835831403192340418920702955983736518'], ['5530633663261076401991476732492823958498424190092130034428794483809683814066', '7329887774871065098982868300009985936454846478611056470378416954485878214482', '3353587650054471857401252634368133543021124664124883374002207721727589841303', '1741411346621014044388859227097668085472131141792221203341385552161352846014'], ['12682682348939714958864453880077873929882203326502593225430981648651948463398', '20379707051743256175904269586016286971490020204076805875524506439416639329907', '16811346477625730398528671456289262259996873366493986592742083158387873456376', '9976708960527914864502062550444624938612985554656243208643578621997383622723'], ['19042942414896350609531483707153495810991189783172640294985021040674851157731', '10426009359863634826079386187454547533457017445252208331971688306884321358956', '15249417476265648827992783785583738689620227681576925045534231516835920969105', '10399204483067264333509250776466870307808029190850175365439616795235750241600']]}} [2023-09-07 18:03:13,666 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getSecretShareV2', 'params': {'polyName': 'POLY:SCHAIN_ID:18860195493846376099765944407454631582519828060563062387235736098366005:NODE_ID:14:DKG_ID:3', 'n': 16, 't': 11, 'publicKeys': ['0dac4abf55c382485f63b112f1d381269c5c30ac8ac797b1da9f025a960506b767442c5b443dfc8cc69a7d12bbf4b9c745c8cb7a41f2a6c07c6ad9923d74c1a5', '6a361515e86bf759bf3e6ab5ff876a428570410f57788ee8bc135e7111f94e9feaee770d62f322aee9a3b19acb416b8b6c5228df3185b3a2b2e4d5688aed5a73', '60edf27e91cae9219f16532e9959544e3b3ae4fece0afbad34ea83ddc44581736efc51b60cadb8ebb20a9764dc9c6471744f55caa9642f6310e46b97ff8491bb', 'cf5ac10babd2be5e01006cf03a675b244fb5fcb1960f608178c0c9aaa61d155d247158d55a8a97f33e2df20b874121730aadda33d0241729d3be32c660462775', 'ce2c88352efc9ceddddcca42bd528e76f772a528ae2ea12a34d72062b2d6ce86f6da04711f9d8de530c50f3f053baca939609419f263ffee324f6a11f450cc00', 'c9d428e666721c70ec30328ea62efb245e26aaff4b560428c4b8f2c7f3d598b870ba1161a1d1f22883fe6c52d7cac874c68d292c294f92bc990838e4734316fe', 'ce60731bf92f17420db5703b2d9c1f410f3f7a14096c20681e3bd3f372cc5ce98b34120b819091993362efa7f121f611a82697019c120c7e43cba90016f3d3c0', 'c3771ddbfd86eaea3eb95cb491cd57de7b0246da912fd3cb00d26c545baf35298b7d53ca817f5033a8e756e8b4baaa0fe00813e48ae1ffece525530b5a36fefa', 'e05064f22cf9c4ae3103e362aadbbc788b8044d8e8198bd9c3aceb836c716f98ba8a3ae9881e33abd81bfba22e77dde561feaa283c8e77960dc5b8f5a620cd81', '1dc327e01eca315119db6eeddbc9216a9069584e2e5185485f3b6aea894dc79a65db549a6e2b4943488df48c527a1860eaaea7b07b16e25447a1a1e247a58628', 'c549e448f3e779a6f48e1e555a891ef721934c67b6d6be3aa387d4ba707c3d237688788cce3ff77a5731a85d2d35fdfcd99a721a59f1ad8ca28efcca61663705', '53265d123f2f1b6462df6b92141256a7dbfa002b87241147b16591ae037756ac4e77ba24499a097995b86c0c87e87d902211953cd6a616b89b04c38714970d8a', 'd40aa8bfbaa0d621910012f05afeda8302662bec1a862f67c092e972a968be66da826501acab258dab7fc2ad6d9873d3596adefe1bb38f47e11b6e73a36482b7', '49351716ba3fe96d9fa4c1357a6039fffbff32ceb97cb05ee1e318275299a413bdfc780b8cd12d5a36ba0260d41853de742be925eb0df043a5a8ab7f54de34ec', '656dcde97a1244e245e36dae1c6c1d4e418fedf94a09059057118e1a3a96d5b676ddd2f44cfc4faf5f820363f2d9843443c3cc262d0d48b2d633837e5eb369d9', '22753df837fa486d41434916b76ca1c61d91f367fdda3654eb813b6fcac423fb8c4aa487203e1771c690c337073d52b52744d4d2cf5d7c0d579145c94e943a43']}} [2023-09-07 18:03:13,959 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'secretShare': 'cffd12aa9b2b35571ee4ce8aa5b18757e57481825e50812df6...', 'status': 0}} [2023-09-07 18:03:14,278 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.transactions.tools:46 - Dry run tx: broadcast, sender: 0x523E13D962A7614752eFa3dB62e0466dF0d9E89D, wallet: RedisWalletAdapter, value: 0, [2023-09-07 18:03:15,074 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.transactions.tools:60 - Estimated gas for broadcast: 624152 [2023-09-07 18:03:15,248 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.transactions.tools:126 - Tx: broadcast, Fields: {'chainId': 5, 'gas': 624152, 'nonce': 315, 'value': 0, 'gasPrice': 10000000000, 'type': 1, 'to': '0x69E2C9576B36c8d7938Ba45a1C909B67EC53048E', 'data': '0x7d93019b000002bb9043962a56bacebbde03191f32a67a4b888426ac09f83548ae92e235000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000620000000000000000000000000000000000000000000000000000000000000000b122b894f026b22d00110ecd00e4719ec5581df04b0012a498c27f96635f23a1f00959eac69a43cf5c07891211dda11ccae8316f1850f79c724448f7f1cac7d4d263cc94b2665747e97c13f5f47288ac0dba2b44143d5c7634767618745d22477226001f9d18f23abb38b62847a03b547203a261c256de08085795e4a376dead8227422f320406ce57a95846c7f0bf411e6b139254511b62048205aaa2e19a07211a9156a8c85fc5294ea25c270ece4917984ed10ecfd4a069106172b5c35c11f0172840ed94fefd09109464bee61a145de6d88822f3c3acbb20e93a3c88a3ab529db44d146e6d38a14d5f346f8ccd1411aa558257c10af12e026be176734f5730335fced0e246b591779236310246e2770931e59a8dc4e409f5623e4c609e664082411a856fefd18eb99e858826cc3efebdda279e7ada93f2a28b58899dea9122c6f63aa3c3388452a2cd2095ba98c20616acd9eed6fa92dfdac1b431d2e1dd42fdd87b0128ef52d0e4563d251fb6631a27ab18e2824dd503ab05b1c1202cfaf27584c28216ba74780c0bfb9b524fc350caf92b030f65ccc7314771b2c69fa6430104f4833257b2da0ff7f4f6912d52b6176fcd92df80fb24d0074fcaccbf7911cfe42189a65d1e4a7603a04f2e17c2651ec6c96ee421b191b9fa65d186a2a5e0912140a2e3034275059d56893a30f386a1dcbc34ef6f691bf631365a9e699a4122fbee179c7b231b455abb0898a0226e1ce1fa23aa9d4e49e542b236670e047283762984c62b917ef9a52c4f2b18beb58d6512515f1e521a130668afba8b7bf0105de2465ad8165e10061f513d247febcb5cee7ad21fce3b726bf40b03743bc0026d36a1618cd323178b1acf85a9976f3237a3d39e071587638307c2aef81391ffd2eb6fd23d8379689e97eea0fc64041361f33a8f103fd5059fc3f85b08892094b3ae2c92ff5ef0e5e6f1b9d0e268ad8894e952668d1bfaf13b654c46cd6bf27c9c93acd0a16d5a3c50d149f7192000747dceb72b41ab0a8b6b90e1540f8e826c8d1bbf838b7af6e5091078f12b66e5d373a4ddcfa99294477aed853bb62ab0a48eff2b64fe23ab0cb1a1ca807b5b4e3ec5f14b3c1030ed0c6f68a9a6677322516f2da08c754ff829f1db7341dd12c768a5594b9b0472e876b835da6d8a225120481e1209ee88940bdc367c7c00ed20564fec85c3782847f5b21f8f653b68e273da697eb8f68aaabe04d52ab2fd8230e41c76e76718ce39759d8eef4e382202efd2deacf683ccb9332a05992514b92cbfbb5fc4b64b5337cbc184d598a516419f51bb15dc87d19f173e7c5b0a0c285d895f92872b9673fadeb0ac2d07ff5d319d4ffca7811330714d4cd3c21535038d1bb6b35926f6d0d89d54c9c0ad6a1d011e63a4157a416f407c08a574863d5492e3e65a7c406a53f8801793690d80ec60c3a3a4c0279a923ed7e7dda63090fd276a6b6c1f84710a613d33a8ed670b6b2103491c82f1ad0b05445743b9c1518e2e508dee90d980dfac98c5a99a5aecf52076a1037626f863e963d4badfcb280b86e214b1b1bf3fcd66a3d65aac8a7799703d99a98c7131b777bd4de909a3fab53618d6fdb3273668e8e61be9061e60abe1c0a24d1c24f1fc45b3b27cfa4be3d2aed635e0331fe1544062803a7af3915262d0e8163749118d6ae8fc10c39110255ca694c4aea2132315a1197e50b9b1673252ae2898fbc490e6cb154fe6fcd583d9933011ac5760a6b022206b15922a0f8160e9dfe666bf06ff0ed978cd48693b8ee60f40443cf2179167d7ef13a7320432a19ec675340416a5d5f90347f40ceb4870a85d8b6e858c0e35e97a8b9da76e3170ce9847282ca2a73cc07ddedf1476fd4fae3d99460e613ec0456ac199a346c21b6dd4af940feb25050c10e6a1f61cb01e498b05acc44aae80c78c8364f7d9116fdbdbc4ef9663f2b1999e3c9c53040a0e71e3d83adde59785dffdb13a7e5400000000000000000000000000000000000000000000000000000000000000010629ea3f8fba8d4ad6eedd952aea6eb2a150a1d3a8f68c3d1c46f51fa0810974339f4bc8e2de99697166d0fb2b1def234031da4a2492db13935c4cde59cf92932cffd12aa9b2b35571ee4ce8aa5b18757e57481825e50812df6890bf1bed9920f520d40b914d7ef6e197a2e3431d81d49fe1cfb6d35e5deab015a47b64733616f6810032267125297f74e58227caf1c68016f2e8e555b3318a1605f1fd3fe24f811d1d1958e0938765d125a24630010f730fcaac0e2332ca545bece92faa0d49093c6292e13e877e5b51cc16b921f9b77db7ff3b6b8d1249564af3bae8b7de62c9506f19b20f6698d007b1cf34709044ce316bd649b4797170a6af90fff4698ccf9624a59fbe40f2b1d3c8fa00793748f9a5b84c2fcd47e22862f24faacf02c552de5e80949ee6e65aaab36e29c510872a323e736d759366cc5aea5146b02a2caa795674ee190244c479cd038ee2a5f16b40fea9df146ba5bee9a02aff9556c73fa44267507c80dd39188fd466c75dfeed9e584a64b7d706ea10afced0318b18599b908c9f039cb4df62bc446bcadea8c60c937bb710343f7add76445fb625b85655aaf934fcdaab5a3769eead853d755554e342d65115f7cdf8eda8fda369585dd368c56a0920517fdcc497076cc5497d48f113c5900138f8546098635a179e3951b327001b386f425b4551b9ff9aaf35081025688f7ac7075a0e60e28c9b90bf75d906562b1735b8eeb24a32e7e531f92f317a5cf08ad89fcf7624e94c9367124d9cf9f050562b79676e86421c39084add3fbde42920c1788dabeca37ed80818049f086196a22be58630fbfd76ad4cb2a1999a60a8c0c37ee3a9a3a47b6374aa169fc1ad2169ce2e3d1c998e84a06a2f99360653c11c71a7bbc39395375631eb04b69978da8cd50de59c0ada1840a775d87279c79856c19c599a443e1567790a26328ebb179f118a67812e88e43097c4f103abef11d9bba29cf22ff3cbbe380468ade8baa32340f96f93fe66cca234eb2f8c679891a90253115b7f9b24ae34759a4dd8c508d4619d1b8e1609d8c17690fb248b6b30b030912659be4a1427666c73268269a319d047a3fbc7747bff2beee608bfcb5d7b6dd16229f86ab82685e0cc87a6f50b1c553f69abf16e7366b07892dcfa921294fefecbc5df426e0be7637ae478ab75379ab4c0dc1a9982bab542c8c1542b4eee36708c66d286312aaefedf3a2ba56e488db72fd7db06d629a3854087652b95dd3a034dab3dffce0711223fd6ffa9aafe64b661470d45cbbef0120e6e62a9745aae4b840789770dc9c78725b0d81f9d93c7fd50a3846715cb091f37d1912f10e975e2f5b8d8c1dd0e5847a0560a809432a7825bd0b8638376bacb020d6ab17be735e18c6d0c0feaca2c84aa0783a1e2b67463c3ad84e1fc8a4bb1d09682a257a158f09486dbac4c39769ef3da765346612be828b2d2b9d60181f2ad8d250fa587c59d7fe94df46d992d662d86ac9b0f8f77fa8ae704215292dd91a1c6583db86afed0a21ff0c67eb896b1908934e4b1f693403dd04f7a16ec411f322012214f976774c8d25a94af22d20988435fd0deb5fcf5c5466b1fec93d4a5d3ba35e5627e537c51ff85b37e61fea004efbaca628c0486d4d6bd0f3dd618507807c0c67dbf6377992b4e668050c5ed1f8323eb2377b8e4a3741d4cec26136d6e28599367d123ca4044264a1b167a76c34e0b13bbb8cad8cd079c38b0a0762b53fa1b2858ea3abaa3074716be2ab0d6676d6b4ce723a7809fbb401c13a8ba4c60e514c70dd89048cce3ee0dc57dd06c27fe17e89ef7bcd78fa424af101e372c91023137748eb94ddddd3ca00d372396bfaebe8b51e3791bda7ee75f3df599e949b05c857d6a0ebd763740e16022497f003eb74b8a82cca3241cbc331d14be8789db2f65e37b5a0e7bf25dbdabee4782c90f7d5b286ca2d4fad9ae26c440394d98cc7773429d27b7940392c2a6d0bd4ec76e8ef165b475d47c9ad85b9a2b1a55c8f18ab17dae2d32b7951377cb126b7a53e3785f2429fc0c1bfa462c209fa0588729f65456c292c31504c3f4ac92d6d2fa4598a5f5e7bec4259fd7a2790dfaf8cd9b6d0fdf5473570390220e54c98ef915d509af4f392935d60c4088cbe77321c2a1806f3797a3b3e61501e0b457c93'}, [2023-09-07 18:03:15,248 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.wallets.redis_wallet:135 - Sending {'chainId': 5, 'gas': 624152, 'nonce': 315, 'value': 0, 'gasPrice': 10000000000, 'type': 1, 'to': '0x69E2C9576B36c8d7938Ba45a1C909B67EC53048E', 'data': '0x7d93019b000002bb9043962a56bacebbde03191f32a67a4b888426ac09f83548ae92e235000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000620000000000000000000000000000000000000000000000000000000000000000b122b894f026b22d00110ecd00e4719ec5581df04b0012a498c27f96635f23a1f00959eac69a43cf5c07891211dda11ccae8316f1850f79c724448f7f1cac7d4d263cc94b2665747e97c13f5f47288ac0dba2b44143d5c7634767618745d22477226001f9d18f23abb38b62847a03b547203a261c256de08085795e4a376dead8227422f320406ce57a95846c7f0bf411e6b139254511b62048205aaa2e19a07211a9156a8c85fc5294ea25c270ece4917984ed10ecfd4a069106172b5c35c11f0172840ed94fefd09109464bee61a145de6d88822f3c3acbb20e93a3c88a3ab529db44d146e6d38a14d5f346f8ccd1411aa558257c10af12e026be176734f5730335fced0e246b591779236310246e2770931e59a8dc4e409f5623e4c609e664082411a856fefd18eb99e858826cc3efebdda279e7ada93f2a28b58899dea9122c6f63aa3c3388452a2cd2095ba98c20616acd9eed6fa92dfdac1b431d2e1dd42fdd87b0128ef52d0e4563d251fb6631a27ab18e2824dd503ab05b1c1202cfaf27584c28216ba74780c0bfb9b524fc350caf92b030f65ccc7314771b2c69fa6430104f4833257b2da0ff7f4f6912d52b6176fcd92df80fb24d0074fcaccbf7911cfe42189a65d1e4a7603a04f2e17c2651ec6c96ee421b191b9fa65d186a2a5e0912140a2e3034275059d56893a30f386a1dcbc34ef6f691bf631365a9e699a4122fbee179c7b231b455abb0898a0226e1ce1fa23aa9d4e49e542b236670e047283762984c62b917ef9a52c4f2b18beb58d6512515f1e521a130668afba8b7bf0105de2465ad8165e10061f513d247febcb5cee7ad21fce3b726bf40b03743bc0026d36a1618cd323178b1acf85a9976f3237a3d39e071587638307c2aef81391ffd2eb6fd23d8379689e97eea0fc64041361f33a8f103fd5059fc3f85b08892094b3ae2c92ff5ef0e5e6f1b9d0e268ad8894e952668d1bfaf13b654c46cd6bf27c9c93acd0a16d5a3c50d149f7192000747dceb72b41ab0a8b6b90e1540f8e826c8d1bbf838b7af6e5091078f12b66e5d373a4ddcfa99294477aed853bb62ab0a48eff2b64fe23ab0cb1a1ca807b5b4e3ec5f14b3c1030ed0c6f68a9a6677322516f2da08c754ff829f1db7341dd12c768a5594b9b0472e876b835da6d8a225120481e1209ee88940bdc367c7c00ed20564fec85c3782847f5b21f8f653b68e273da697eb8f68aaabe04d52ab2fd8230e41c76e76718ce39759d8eef4e382202efd2deacf683ccb9332a05992514b92cbfbb5fc4b64b5337cbc184d598a516419f51bb15dc87d19f173e7c5b0a0c285d895f92872b9673fadeb0ac2d07ff5d319d4ffca7811330714d4cd3c21535038d1bb6b35926f6d0d89d54c9c0ad6a1d011e63a4157a416f407c08a574863d5492e3e65a7c406a53f8801793690d80ec60c3a3a4c0279a923ed7e7dda63090fd276a6b6c1f84710a613d33a8ed670b6b2103491c82f1ad0b05445743b9c1518e2e508dee90d980dfac98c5a99a5aecf52076a1037626f863e963d4badfcb280b86e214b1b1bf3fcd66a3d65aac8a7799703d99a98c7131b777bd4de909a3fab53618d6fdb3273668e8e61be9061e60abe1c0a24d1c24f1fc45b3b27cfa4be3d2aed635e0331fe1544062803a7af3915262d0e8163749118d6ae8fc10c39110255ca694c4aea2132315a1197e50b9b1673252ae2898fbc490e6cb154fe6fcd583d9933011ac5760a6b022206b15922a0f8160e9dfe666bf06ff0ed978cd48693b8ee60f40443cf2179167d7ef13a7320432a19ec675340416a5d5f90347f40ceb4870a85d8b6e858c0e35e97a8b9da76e3170ce9847282ca2a73cc07ddedf1476fd4fae3d99460e613ec0456ac199a346c21b6dd4af940feb25050c10e6a1f61cb01e498b05acc44aae80c78c8364f7d9116fdbdbc4ef9663f2b1999e3c9c53040a0e71e3d83adde59785dffdb13a7e5400000000000000000000000000000000000000000000000000000000000000010629ea3f8fba8d4ad6eedd952aea6eb2a150a1d3a8f68c3d1c46f51fa0810974339f4bc8e2de99697166d0fb2b1def234031da4a2492db13935c4cde59cf92932cffd12aa9b2b35571ee4ce8aa5b18757e57481825e50812df6890bf1bed9920f520d40b914d7ef6e197a2e3431d81d49fe1cfb6d35e5deab015a47b64733616f6810032267125297f74e58227caf1c68016f2e8e555b3318a1605f1fd3fe24f811d1d1958e0938765d125a24630010f730fcaac0e2332ca545bece92faa0d49093c6292e13e877e5b51cc16b921f9b77db7ff3b6b8d1249564af3bae8b7de62c9506f19b20f6698d007b1cf34709044ce316bd649b4797170a6af90fff4698ccf9624a59fbe40f2b1d3c8fa00793748f9a5b84c2fcd47e22862f24faacf02c552de5e80949ee6e65aaab36e29c510872a323e736d759366cc5aea5146b02a2caa795674ee190244c479cd038ee2a5f16b40fea9df146ba5bee9a02aff9556c73fa44267507c80dd39188fd466c75dfeed9e584a64b7d706ea10afced0318b18599b908c9f039cb4df62bc446bcadea8c60c937bb710343f7add76445fb625b85655aaf934fcdaab5a3769eead853d755554e342d65115f7cdf8eda8fda369585dd368c56a0920517fdcc497076cc5497d48f113c5900138f8546098635a179e3951b327001b386f425b4551b9ff9aaf35081025688f7ac7075a0e60e28c9b90bf75d906562b1735b8eeb24a32e7e531f92f317a5cf08ad89fcf7624e94c9367124d9cf9f050562b79676e86421c39084add3fbde42920c1788dabeca37ed80818049f086196a22be58630fbfd76ad4cb2a1999a60a8c0c37ee3a9a3a47b6374aa169fc1ad2169ce2e3d1c998e84a06a2f99360653c11c71a7bbc39395375631eb04b69978da8cd50de59c0ada1840a775d87279c79856c19c599a443e1567790a26328ebb179f118a67812e88e43097c4f103abef11d9bba29cf22ff3cbbe380468ade8baa32340f96f93fe66cca234eb2f8c679891a90253115b7f9b24ae34759a4dd8c508d4619d1b8e1609d8c17690fb248b6b30b030912659be4a1427666c73268269a319d047a3fbc7747bff2beee608bfcb5d7b6dd16229f86ab82685e0cc87a6f50b1c553f69abf16e7366b07892dcfa921294fefecbc5df426e0be7637ae478ab75379ab4c0dc1a9982bab542c8c1542b4eee36708c66d286312aaefedf3a2ba56e488db72fd7db06d629a3854087652b95dd3a034dab3dffce0711223fd6ffa9aafe64b661470d45cbbef0120e6e62a9745aae4b840789770dc9c78725b0d81f9d93c7fd50a3846715cb091f37d1912f10e975e2f5b8d8c1dd0e5847a0560a809432a7825bd0b8638376bacb020d6ab17be735e18c6d0c0feaca2c84aa0783a1e2b67463c3ad84e1fc8a4bb1d09682a257a158f09486dbac4c39769ef3da765346612be828b2d2b9d60181f2ad8d250fa587c59d7fe94df46d992d662d86ac9b0f8f77fa8ae704215292dd91a1c6583db86afed0a21ff0c67eb896b1908934e4b1f693403dd04f7a16ec411f322012214f976774c8d25a94af22d20988435fd0deb5fcf5c5466b1fec93d4a5d3ba35e5627e537c51ff85b37e61fea004efbaca628c0486d4d6bd0f3dd618507807c0c67dbf6377992b4e668050c5ed1f8323eb2377b8e4a3741d4cec26136d6e28599367d123ca4044264a1b167a76c34e0b13bbb8cad8cd079c38b0a0762b53fa1b2858ea3abaa3074716be2ab0d6676d6b4ce723a7809fbb401c13a8ba4c60e514c70dd89048cce3ee0dc57dd06c27fe17e89ef7bcd78fa424af101e372c91023137748eb94ddddd3ca00d372396bfaebe8b51e3791bda7ee75f3df599e949b05c857d6a0ebd763740e16022497f003eb74b8a82cca3241cbc331d14be8789db2f65e37b5a0e7bf25dbdabee4782c90f7d5b286ca2d4fad9ae26c440394d98cc7773429d27b7940392c2a6d0bd4ec76e8ef165b475d47c9ad85b9a2b1a55c8f18ab17dae2d32b7951377cb126b7a53e3785f2429fc0c1bfa462c209fa0588729f65456c292c31504c3f4ac92d6d2fa4598a5f5e7bec4259fd7a2790dfaf8cd9b6d0fdf5473570390220e54c98ef915d509af4f392935d60c4088cbe77321c2a1806f3797a3b3e61501e0b457c93'} to redis pool, method: dkg.broadcast [2023-09-07 18:03:15,249 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.wallets.redis_wallet:145 - Adding tx b'tx-e92b39b5bc4398fb' to the pool [2023-09-07 18:03:15,249 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.wallets.redis_wallet:147 - Saving tx b'tx-e92b39b5bc4398fb' record: b'{"status": "PROPOSED", "score": 51694109795, "multiplier": null, "tx_hash": null, "method": "dkg.broadcast", "meta": null, "chainId": 5, "gas": null, "nonce": 315, "value": 0, "gasPrice": 10000000000, "type": 1, "to": "0x69E2C9576B36c8d7938Ba45a1C909B67EC53048E", "data": "0x7d93019b000002bb9043962a56bacebbde03191f32a67a4b888426ac09f83548ae92e235000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000620000000000000000000000000000000000000000000000000000000000000000b122b894f026b22d00110ecd00e4719ec5581df04b0012a498c27f96635f23a1f00959eac69a43cf5c07891211dda11ccae8316f1850f79c724448f7f1cac7d4d263cc94b2665747e97c13f5f47288ac0dba2b44143d5c7634767618745d22477226001f9d18f23abb38b62847a03b547203a261c256de08085795e4a376dead8227422f320406ce57a95846c7f0bf411e6b139254511b62048205aaa2e19a07211a9156a8c85fc5294ea25c270ece4917984ed10ecfd4a069106172b5c35c11f0172840ed94fefd09109464bee61a145de6d88822f3c3acbb20e93a3c88a3ab529db44d146e6d38a14d5f346f8ccd1411aa558257c10af12e026be176734f5730335fced0e246b591779236310246e2770931e59a8dc4e409f5623e4c609e664082411a856fefd18eb99e858826cc3efebdda279e7ada93f2a28b58899dea9122c6f63aa3c3388452a2cd2095ba98c20616acd9eed6fa92dfdac1b431d2e1dd42fdd87b0128ef52d0e4563d251fb6631a27ab18e2824dd503ab05b1c1202cfaf27584c28216ba74780c0bfb9b524fc350caf92b030f65ccc7314771b2c69fa6430104f4833257b2da0ff7f4f6912d52b6176fcd92df80fb24d0074fcaccbf7911cfe42189a65d1e4a7603a04f2e17c2651ec6c96ee421b191b9fa65d186a2a5e0912140a2e3034275059d56893a30f386a1dcbc34ef6f691bf631365a9e699a4122fbee179c7b231b455abb0898a0226e1ce1fa23aa9d4e49e542b236670e047283762984c62b917ef9a52c4f2b18beb58d6512515f1e521a130668afba8b7bf0105de2465ad8165e10061f513d247febcb5cee7ad21fce3b726bf40b03743bc0026d36a1618cd323178b1acf85a9976f3237a3d39e071587638307c2aef81391ffd2eb6fd23d8379689e97eea0fc64041361f33a8f103fd5059fc3f85b08892094b3ae2c92ff5ef0e5e6f1b9d0e268ad8894e952668d1bfaf13b654c46cd6bf27c9c93acd0a16d5a3c50d149f7192000747dceb72b41ab0a8b6b90e1540f8e826c8d1bbf838b7af6e5091078f12b66e5d373a4ddcfa99294477aed853bb62ab0a48eff2b64fe23ab0cb1a1ca807b5b4e3ec5f14b3c1030ed0c6f68a9a6677322516f2da08c754ff829f1db7341dd12c768a5594b9b0472e876b835da6d8a225120481e1209ee88940bdc367c7c00ed20564fec85c3782847f5b21f8f653b68e273da697eb8f68aaabe04d52ab2fd8230e41c76e76718ce39759d8eef4e382202efd2deacf683ccb9332a05992514b92cbfbb5fc4b64b5337cbc184d598a516419f51bb15dc87d19f173e7c5b0a0c285d895f92872b9673fadeb0ac2d07ff5d319d4ffca7811330714d4cd3c21535038d1bb6b35926f6d0d89d54c9c0ad6a1d011e63a4157a416f407c08a574863d5492e3e65a7c406a53f8801793690d80ec60c3a3a4c0279a923ed7e7dda63090fd276a6b6c1f84710a613d33a8ed670b6b2103491c82f1ad0b05445743b9c1518e2e508dee90d980dfac98c5a99a5aecf52076a1037626f863e963d4badfcb280b86e214b1b1bf3fcd66a3d65aac8a7799703d99a98c7131b777bd4de909a3fab53618d6fdb3273668e8e61be9061e60abe1c0a24d1c24f1fc45b3b27cfa4be3d2aed635e0331fe1544062803a7af3915262d0e8163749118d6ae8fc10c39110255ca694c4aea2132315a1197e50b9b1673252ae2898fbc490e6cb154fe6fcd583d9933011ac5760a6b022206b15922a0f8160e9dfe666bf06ff0ed978cd48693b8ee60f40443cf2179167d7ef13a7320432a19ec675340416a5d5f90347f40ceb4870a85d8b6e858c0e35e97a8b9da76e3170ce9847282ca2a73cc07ddedf1476fd4fae3d99460e613ec0456ac199a346c21b6dd4af940feb25050c10e6a1f61cb01e498b05acc44aae80c78c8364f7d9116fdbdbc4ef9663f2b1999e3c9c53040a0e71e3d83adde59785dffdb13a7e5400000000000000000000000000000000000000000000000000000000000000010629ea3f8fba8d4ad6eedd952aea6eb2a150a1d3a8f68c3d1c46f51fa0810974339f4bc8e2de99697166d0fb2b1def234031da4a2492db13935c4cde59cf92932cffd12aa9b2b35571ee4ce8aa5b18757e57481825e50812df6890bf1bed9920f520d40b914d7ef6e197a2e3431d81d49fe1cfb6d35e5deab015a47b64733616f6810032267125297f74e58227caf1c68016f2e8e555b3318a1605f1fd3fe24f811d1d1958e0938765d125a24630010f730fcaac0e2332ca545bece92faa0d49093c6292e13e877e5b51cc16b921f9b77db7ff3b6b8d1249564af3bae8b7de62c9506f19b20f6698d007b1cf34709044ce316bd649b4797170a6af90fff4698ccf9624a59fbe40f2b1d3c8fa00793748f9a5b84c2fcd47e22862f24faacf02c552de5e80949ee6e65aaab36e29c510872a323e736d759366cc5aea5146b02a2caa795674ee190244c479cd038ee2a5f16b40fea9df146ba5bee9a02aff9556c73fa44267507c80dd39188fd466c75dfeed9e584a64b7d706ea10afced0318b18599b908c9f039cb4df62bc446bcadea8c60c937bb710343f7add76445fb625b85655aaf934fcdaab5a3769eead853d755554e342d65115f7cdf8eda8fda369585dd368c56a0920517fdcc497076cc5497d48f113c5900138f8546098635a179e3951b327001b386f425b4551b9ff9aaf35081025688f7ac7075a0e60e28c9b90bf75d906562b1735b8eeb24a32e7e531f92f317a5cf08ad89fcf7624e94c9367124d9cf9f050562b79676e86421c39084add3fbde42920c1788dabeca37ed80818049f086196a22be58630fbfd76ad4cb2a1999a60a8c0c37ee3a9a3a47b6374aa169fc1ad2169ce2e3d1c998e84a06a2f99360653c11c71a7bbc39395375631eb04b69978da8cd50de59c0ada1840a775d87279c79856c19c599a443e1567790a26328ebb179f118a67812e88e43097c4f103abef11d9bba29cf22ff3cbbe380468ade8baa32340f96f93fe66cca234eb2f8c679891a90253115b7f9b24ae34759a4dd8c508d4619d1b8e1609d8c17690fb248b6b30b030912659be4a1427666c73268269a319d047a3fbc7747bff2beee608bfcb5d7b6dd16229f86ab82685e0cc87a6f50b1c553f69abf16e7366b07892dcfa921294fefecbc5df426e0be7637ae478ab75379ab4c0dc1a9982bab542c8c1542b4eee36708c66d286312aaefedf3a2ba56e488db72fd7db06d629a3854087652b95dd3a034dab3dffce0711223fd6ffa9aafe64b661470d45cbbef0120e6e62a9745aae4b840789770dc9c78725b0d81f9d93c7fd50a3846715cb091f37d1912f10e975e2f5b8d8c1dd0e5847a0560a809432a7825bd0b8638376bacb020d6ab17be735e18c6d0c0feaca2c84aa0783a1e2b67463c3ad84e1fc8a4bb1d09682a257a158f09486dbac4c39769ef3da765346612be828b2d2b9d60181f2ad8d250fa587c59d7fe94df46d992d662d86ac9b0f8f77fa8ae704215292dd91a1c6583db86afed0a21ff0c67eb896b1908934e4b1f693403dd04f7a16ec411f322012214f976774c8d25a94af22d20988435fd0deb5fcf5c5466b1fec93d4a5d3ba35e5627e537c51ff85b37e61fea004efbaca628c0486d4d6bd0f3dd618507807c0c67dbf6377992b4e668050c5ed1f8323eb2377b8e4a3741d4cec26136d6e28599367d123ca4044264a1b167a76c34e0b13bbb8cad8cd079c38b0a0762b53fa1b2858ea3abaa3074716be2ab0d6676d6b4ce723a7809fbb401c13a8ba4c60e514c70dd89048cce3ee0dc57dd06c27fe17e89ef7bcd78fa424af101e372c91023137748eb94ddddd3ca00d372396bfaebe8b51e3791bda7ee75f3df599e949b05c857d6a0ebd763740e16022497f003eb74b8a82cca3241cbc331d14be8789db2f65e37b5a0e7bf25dbdabee4782c90f7d5b286ca2d4fad9ae26c440394d98cc7773429d27b7940392c2a6d0bd4ec76e8ef165b475d47c9ad85b9a2b1a55c8f18ab17dae2d32b7951377cb126b7a53e3785f2429fc0c1bfa462c209fa0588729f65456c292c31504c3f4ac92d6d2fa4598a5f5e7bec4259fd7a2790dfaf8cd9b6d0fdf5473570390220e54c98ef915d509af4f392935d60c4088cbe77321c2a1806f3797a3b3e61501e0b457c93"}' [2023-09-07 18:03:15,710 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109768, schain_finish_ts: 1693742304 [2023-09-07 18:03:16,925 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:03:16,925 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:03:16,926 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:03:16,926 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 18:03:16,926 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 18:03:16,927 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:03:16,931 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:03:16,931 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 18:03:16,964 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:03:17,316 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:17,642 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:17,791 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:03:17,791 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2569 [2023-09-07 18:03:17,792 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:03:17,792 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:03:17,793 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:03:17,793 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:03:17,793 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:03:17,800 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:03:17,802 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:03:17.802682 [2023-09-07 18:03:18,042 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:18,070 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:03:18,105 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:03:18,108 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:03:18,123 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:03:18,124 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109798.068505, 'te': 1694109798.0752304, 'initial_status': True} ====================================================================================================  [2023-09-07 18:03:18,126 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:03:18.126031 [2023-09-07 18:03:18,130 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:03:18,130 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:03:18,821 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 18:03:18,821 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:03:18,824 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:03:18.824427 [2023-09-07 18:03:18,832 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:03:19,612 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109792, schain_finish_ts: 1693412520 [2023-09-07 18:03:20,736 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:03:20,737 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:03:20,737 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:03:20,737 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:03:20,738 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:03:20,738 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:03:20,740 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:03:20,740 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 18:03:20,784 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:03:21,079 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:21,256 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:21,415 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:03:21,418 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2566 [2023-09-07 18:03:21,419 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:03:21,420 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:03:21,420 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:03:21,420 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:03:21,421 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:03:21,431 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:03:21,435 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:03:21.435215 [2023-09-07 18:03:21,756 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:21,787 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:03:21,825 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:03:21,828 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:03:21,836 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:03:21,837 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109801.782724, 'te': 1694109801.7981002, 'initial_status': True} ====================================================================================================  [2023-09-07 18:03:21,839 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:03:21.839789 [2023-09-07 18:03:21,844 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:03:21,845 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:03:22,635 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:03:22,635 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:03:22,638 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:03:22.638886 [2023-09-07 18:03:22,650 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:03:22,650 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:03:22,659 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:03:22,664 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109802.646948, 'te': 1694109802.664335, 'initial_status': True} ====================================================================================================  [2023-09-07 18:03:22,667 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:03:22.667326 [2023-09-07 18:03:22,671 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:03:22,672 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:03:26,807 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 18:03:26,807 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 18:03:29,728 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109792, schain_finish_ts: 1694108304 [2023-09-07 18:03:30,938 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:03:30,938 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:03:30,939 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 18:03:33,726 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109792, schain_finish_ts: 1692637668 [2023-09-07 18:03:34,875 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:03:34,876 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:03:34,876 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:03:34,876 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:03:34,876 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:03:34,877 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:03:34,878 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:03:34,883 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 18:03:34,910 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:03:35,193 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:35,572 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:35,711 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:03:35,712 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2564 [2023-09-07 18:03:35,712 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:03:35,713 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:03:35,713 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:03:35,713 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:03:35,714 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:03:35,720 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:03:35,722 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:03:35.722912 [2023-09-07 18:03:35,879 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:35,914 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:03:35,964 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:03:35,966 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:03:35,973 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:03:35,974 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109815.9120436, 'te': 1694109815.9284775, 'initial_status': True} ====================================================================================================  [2023-09-07 18:03:35,975 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:03:35.975910 [2023-09-07 18:03:35,980 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:03:35,981 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:03:36,293 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109792, schain_finish_ts: 1693742004 [2023-09-07 18:03:36,758 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:03:36,758 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:03:36,760 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:03:36.760938 [2023-09-07 18:03:36,770 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:03:36,771 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:03:36,778 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:03:36,784 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109816.7675061, 'te': 1694109816.7842646, 'initial_status': True} ====================================================================================================  [2023-09-07 18:03:36,787 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:03:36.787578 [2023-09-07 18:03:36,792 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:03:36,793 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:03:37,436 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:03:37,436 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:03:37,437 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:03:37,437 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:03:37,437 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:03:37,438 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 18:03:37,476 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:03:37,784 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:37,990 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:40 - Collecting rotation history for submissive-well-off-rotanev... [2023-09-07 18:03:38,062 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:38,224 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:03:38,225 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2562 [2023-09-07 18:03:38,226 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:03:38,226 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:03:38,226 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:03:38,227 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:03:38,227 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:03:38,234 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:03:38,236 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:03:38.236645 [2023-09-07 18:03:38,351 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 18:03:38,351 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 18:03:38,436 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:38,472 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:03:38,513 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:03:38,516 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:03:38,526 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:03:38,526 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109818.470798, 'te': 1694109818.4782941, 'initial_status': True} ====================================================================================================  [2023-09-07 18:03:38,528 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:03:38.528771 [2023-09-07 18:03:38,533 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:03:38,533 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:03:39,125 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:50 - Rotation data for submissive-well-off-rotanev: Rotation(leaving_node_id=65, new_node_id=65, freeze_until=1694108100, rotation_counter=8) [2023-09-07 18:03:39,896 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109792, schain_finish_ts: 1693742304 [2023-09-07 18:03:40,555 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109792, schain_finish_ts: 0 [2023-09-07 18:03:41,089 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:03:41,095 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:03:41,098 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:03:41,098 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:03:41,099 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:03:41,099 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 18:03:41,125 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:03:41,387 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:41,616 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:41,719 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:03:41,719 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:03:41,719 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 18:03:41,746 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:03:41,747 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2570 [2023-09-07 18:03:41,748 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:03:41,748 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:03:41,748 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:03:41,749 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:03:41,750 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:03:41,760 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:03:41,763 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:03:41.763122 [2023-09-07 18:03:41,977 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:42,015 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:03:42,051 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:03:42,054 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:03:42,063 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:03:42,063 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109822.0080907, 'te': 1694109822.0204213, 'initial_status': True} ====================================================================================================  [2023-09-07 18:03:42,066 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:03:42.066603 [2023-09-07 18:03:42,071 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:03:42,072 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:03:45,799 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109792, schain_finish_ts: 1692377364 [2023-09-07 18:03:46,795 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 18:03:46,921 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:03:46,921 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:03:46,921 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:03:46,922 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:03:46,922 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:03:46,922 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:03:46,924 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:03:46,925 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 18:03:46,952 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:03:46,972 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:03:46,973 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 18:03:46,973 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 18:03:46,974 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 18:03:47,156 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 18:03:47,157 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 18:03:47,159 INFO][8][curly-red-alterf][T_0] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 18:03:47,167 ERROR][8][curly-red-alterf][T_0] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:03:47,168 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:03:47,203 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:47,371 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:47,490 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:03:47,491 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2549 [2023-09-07 18:03:47,491 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:03:47,492 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:03:47,492 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:03:47,492 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:03:47,492 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:03:47,499 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:03:47,509 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:03:47.509436 [2023-09-07 18:03:47,672 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:03:47,704 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:03:47,735 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:03:47,737 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:03:47,745 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:03:47,745 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109827.7024312, 'te': 1694109827.7092128, 'initial_status': True} ====================================================================================================  [2023-09-07 18:03:47,747 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:03:47.747530 [2023-09-07 18:03:47,752 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:03:47,752 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:03:48,734 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:03:48,734 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:03:48,737 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:03:48.737112 [2023-09-07 18:03:48,744 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:03:48,744 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:03:48,750 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:03:48,755 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109828.7425041, 'te': 1694109828.755049, 'initial_status': True} ====================================================================================================  [2023-09-07 18:03:48,757 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:03:48.757910 [2023-09-07 18:03:48,762 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:03:48,762 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:03:56,643 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 63 -> 65 [2023-09-07 18:03:57,297 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109828, schain_finish_ts: 1693412520 [2023-09-07 18:04:00,429 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:04:00,429 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:04:00,431 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:04:00,432 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:04:00,435 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:04:00,436 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:04:00,441 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:04:00,443 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 18:04:00,477 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:04:00,800 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:04:00,997 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:04:01,128 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:04:01,129 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2567 [2023-09-07 18:04:01,130 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:04:01,130 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:04:01,130 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:04:01,131 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:04:01,131 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:04:01,138 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:04:01,140 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:04:01.140784 [2023-09-07 18:04:01,323 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:04:01,359 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:04:01,395 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:04:01,397 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:04:01,406 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:04:01,407 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109841.356246, 'te': 1694109841.3637712, 'initial_status': True} ====================================================================================================  [2023-09-07 18:04:01,408 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:04:01.408970 [2023-09-07 18:04:01,415 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:04:01,416 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:04:06,474 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 18:04:06,650 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:04:06,650 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:04:06,650 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:04:06,651 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 18:04:06,839 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 18:04:06,839 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 18:04:06,845 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 18:04:06,851 ERROR][11][wan-tasty-kitalpha][T_1] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:04:06,852 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:04:07,278 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:04:07,279 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:04:07,281 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:04:07.281120 [2023-09-07 18:04:07,289 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:04:07,289 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:04:07,295 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:04:07,300 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109847.2868278, 'te': 1694109847.3005064, 'initial_status': True} ====================================================================================================  [2023-09-07 18:04:07,302 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:04:07.302289 [2023-09-07 18:04:07,306 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:04:07,306 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:04:08,930 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109840, schain_finish_ts: 0 [2023-09-07 18:04:09,237 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 23 -> 24 [2023-09-07 18:04:09,458 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109840, schain_finish_ts: 1694108304 [2023-09-07 18:04:09,462 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109840, schain_finish_ts: 1692377364 [2023-09-07 18:04:13,620 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109840, schain_finish_ts: 1692637668 [2023-09-07 18:04:14,317 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:04:14,317 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:04:14,318 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 18:04:16,264 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:04:16,265 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:04:16,266 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:04:16,268 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:04:16,269 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:04:16,269 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:04:16,275 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:04:16,276 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 18:04:16,306 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:04:16,367 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:04:16,368 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:04:16,368 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 18:04:16,376 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109840, schain_finish_ts: 1693742004 [2023-09-07 18:04:16,729 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:04:16,804 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109840, schain_finish_ts: 1693742304 [2023-09-07 18:04:16,919 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:04:17,072 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:04:17,073 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2550 [2023-09-07 18:04:17,074 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:04:17,074 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:04:17,074 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:04:17,075 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:04:17,075 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:04:17,083 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:04:17,085 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:04:17.085506 [2023-09-07 18:04:17,311 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:04:17,361 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:04:17,397 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:04:17,399 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:04:17,417 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:04:17,417 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109857.3596098, 'te': 1694109857.3662174, 'initial_status': True} ====================================================================================================  [2023-09-07 18:04:17,420 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:04:17.420578 [2023-09-07 18:04:17,426 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:04:17,427 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:04:18,223 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:04:18,223 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:04:18,223 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:04:18,223 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:04:18,224 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:04:18,225 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:04:18,229 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:04:18,230 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 18:04:18,254 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:04:18,455 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:04:18,689 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:04:18,809 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:04:18,809 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:04:18,809 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:04:18,810 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 18:04:18,810 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 18:04:18,810 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:04:18,812 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:04:18,815 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 18:04:18,838 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:04:18,839 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2565 [2023-09-07 18:04:18,840 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:04:18,840 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:04:18,841 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:04:18,841 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:04:18,842 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:04:18,844 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:04:18,849 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:04:18,852 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:04:18.852424 [2023-09-07 18:04:18,993 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:04:19,036 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:04:19,071 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:04:19,073 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:04:19,081 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:04:19,082 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109859.034699, 'te': 1694109859.0411558, 'initial_status': True} ====================================================================================================  [2023-09-07 18:04:19,084 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:04:19.084005 [2023-09-07 18:04:19,088 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:04:19,089 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:04:19,213 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:04:19,536 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:04:19,660 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:04:19,661 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2563 [2023-09-07 18:04:19,662 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:04:19,662 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:04:19,664 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:04:19,664 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:04:19,664 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:04:19,674 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:04:19,676 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:04:19.676798 [2023-09-07 18:04:19,844 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:04:19,873 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:04:19,906 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:04:19,908 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:04:19,917 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:04:19,918 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109859.8712556, 'te': 1694109859.878259, 'initial_status': True} ====================================================================================================  [2023-09-07 18:04:19,919 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:04:19.919863 [2023-09-07 18:04:19,924 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:04:19,924 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:04:20,293 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:04:20,294 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:04:20,298 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:04:20.298935 [2023-09-07 18:04:20,307 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:04:20,307 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:04:20,312 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:04:20,316 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109860.3057358, 'te': 1694109860.316751, 'initial_status': True} ====================================================================================================  [2023-09-07 18:04:20,319 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:04:20.319449 [2023-09-07 18:04:20,324 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:04:20,324 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:04:20,387 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:04:20,387 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:04:20,387 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:04:20,388 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 18:04:20,388 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 18:04:20,388 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:04:20,390 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:04:20,391 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 18:04:20,416 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:04:20,616 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:04:20,874 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:04:20,985 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:04:20,986 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2571 [2023-09-07 18:04:20,987 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:04:20,987 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:04:20,987 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:04:20,988 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:04:20,988 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:04:20,994 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:04:20,996 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:04:20.996352 [2023-09-07 18:04:21,206 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:04:21,239 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:04:21,275 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:04:21,277 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:04:21,287 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:04:21,288 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109861.2364695, 'te': 1694109861.2441728, 'initial_status': True} ====================================================================================================  [2023-09-07 18:04:21,290 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:04:21.290824 [2023-09-07 18:04:21,295 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:04:21,296 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:04:27,542 INFO][1][MainProcess][MainThread] - core.schains.cleaner:65 - Cleaner process started [2023-09-07 18:04:27,586 INFO][30953][cleaner][MainThread] - core.schains.cleaner:110 - Cleaner procedure started. [2023-09-07 18:04:27,751 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 18:04:27,751 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:04:27,754 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:04:27.754066 [2023-09-07 18:04:27,761 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:04:28,350 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:04:28,350 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:04:28,353 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:04:28.353171 [2023-09-07 18:04:28,362 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:04:28,363 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:04:28,369 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:04:28,377 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109868.3609796, 'te': 1694109868.3768377, 'initial_status': True} ====================================================================================================  [2023-09-07 18:04:28,379 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:04:28.379874 [2023-09-07 18:04:28,386 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:04:28,386 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:04:30,610 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 18:04:30,610 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:04:30,613 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:04:30.613080 [2023-09-07 18:04:30,618 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:04:33,679 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 9 -> 64 [2023-09-07 18:04:48,311 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109876, schain_finish_ts: 0 [2023-09-07 18:04:50,268 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109876, schain_finish_ts: 1692637668 [2023-09-07 18:04:55,345 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109888, schain_finish_ts: 1693742304 [2023-09-07 18:04:57,558 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109888, schain_finish_ts: 1694108304 [2023-09-07 18:04:57,564 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109888, schain_finish_ts: 1692377364 [2023-09-07 18:04:58,166 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:04:58,166 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:04:58,166 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 18:04:59,317 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109888, schain_finish_ts: 1693412520 [2023-09-07 18:05:03,280 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:05:03,281 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:05:03,281 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:05:03,281 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:05:03,283 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:05:03,283 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 18:05:03,302 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:05:03,517 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:05:03,708 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:05:03,807 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:05:03,808 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2572 [2023-09-07 18:05:03,808 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:05:03,809 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:05:03,809 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:05:03,809 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:05:03,810 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:05:03,815 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:05:03,817 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:05:03.817801 [2023-09-07 18:05:04,010 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:05:04,048 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:05:04,085 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:05:04,105 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:05:04,114 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:05:04,115 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109904.0471275, 'te': 1694109904.0535643, 'initial_status': True} ====================================================================================================  [2023-09-07 18:05:04,117 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:05:04.116985 [2023-09-07 18:05:04,122 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:05:04,123 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:05:09,140 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109900, schain_finish_ts: 1693742004 [2023-09-07 18:05:11,789 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:05:11,789 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:05:11,790 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:05:11,790 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:05:11,790 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:05:11,791 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:05:11,792 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:05:11,793 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 18:05:11,827 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:05:12,144 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:05:12,329 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:05:12,454 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:05:12,456 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2566 [2023-09-07 18:05:12,456 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:05:12,457 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:05:12,457 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:05:12,457 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:05:12,458 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:05:12,464 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:05:12,467 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:05:12.467559 [2023-09-07 18:05:12,623 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:05:12,654 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:05:12,683 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:05:12,685 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:05:12,693 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:05:12,694 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694109912.6526463, 'te': 1694109912.6583924, 'initial_status': True} ====================================================================================================  [2023-09-07 18:05:12,697 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:05:12.697816 [2023-09-07 18:05:12,702 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:05:12,702 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:05:19,720 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:05:19,721 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:05:19,722 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 18:05:23,122 INFO][30953][cleaner][MainThread] - core.schains.cleaner:116 - sChains on contracts: ['portly-passionate-sirius', 'curly-red-alterf', 'howling-hoedus', 'green-giddy-denebola', 'wan-tasty-kitalpha', 'free-basic-rosy-SKALE', 'submissive-well-off-rotanev'] sChains on node: ['curly-red-alterf', 'free-basic-rosy-SKALE', 'green-giddy-denebola', 'howling-hoedus', 'portly-passionate-sirius', 'submissive-well-off-rotanev', 'wan-tasty-kitalpha'] [2023-09-07 18:05:23,123 INFO][30953][cleaner][MainThread] - core.schains.cleaner:132 - Cleanup procedure finished [2023-09-07 18:05:23,128 INFO][1][MainProcess][MainThread] - core.schains.cleaner:67 - Cleaner process is joined. [2023-09-07 18:05:23,129 INFO][1][MainProcess][MainThread] - core.schains.cleaner:68 - Terminating the process [2023-09-07 18:05:23,130 INFO][1][MainProcess][MainThread] - __main__:70 - Sleeping for 90s after run_cleaner [2023-09-07 18:05:23,180 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:05:23,180 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:05:23,181 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:05:23,181 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:05:23,181 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:05:23,181 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:05:23,181 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:05:23,186 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 18:05:23,215 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:05:23,448 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:05:23,621 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:05:23,735 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:05:23,736 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2551 [2023-09-07 18:05:23,737 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:05:23,737 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:05:23,737 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:05:23,737 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:05:23,738 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:05:23,744 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:05:23,746 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:05:23.746618 [2023-09-07 18:05:24,026 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:05:24,056 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:05:24,092 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:05:24,094 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:05:24,104 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:05:24,104 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109924.053423, 'te': 1694109924.0635715, 'initial_status': True} ====================================================================================================  [2023-09-07 18:05:24,107 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:05:24.106998 [2023-09-07 18:05:24,112 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:05:24,113 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:05:32,335 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:05:32,336 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:05:32,337 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:05:32,338 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:05:32,339 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:05:32,340 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:05:32,345 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:05:32,346 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 18:05:32,373 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:05:32,564 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:05:32,877 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:05:33,027 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:05:33,028 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2568 [2023-09-07 18:05:33,029 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:05:33,029 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:05:33,029 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:05:33,030 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:05:33,030 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:05:33,036 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:05:33,039 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:05:33.039425 [2023-09-07 18:05:33,213 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:05:33,240 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:05:33,276 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:05:33,278 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:05:33,286 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:05:33,286 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694109933.2383626, 'te': 1694109933.2446048, 'initial_status': True} ====================================================================================================  [2023-09-07 18:05:33,288 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:05:33.288320 [2023-09-07 18:05:33,295 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:05:33,296 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:05:40,528 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.transactions.tools:190 - Tx broadcast completed after 1/3 retries [2023-09-07 18:05:40,528 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:230 - sChain: free-basic-rosy-SKALE. Everything is sent from 14 node [2023-09-07 18:05:43,474 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:104 - sChain free-basic-rosy-SKALE: trying to receive broadcasted data,1512 seconds left [2023-09-07 18:05:47,115 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:05:47,115 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:05:47,115 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:05:47,116 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:05:47,116 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:05:47,116 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 18:05:47,147 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:05:47,428 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:05:47,562 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:05:47,674 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:05:47,675 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2564 [2023-09-07 18:05:47,676 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:05:47,676 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:05:47,676 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:05:47,677 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:05:47,677 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:05:47,684 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:05:47,686 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:05:47.686073 [2023-09-07 18:05:47,834 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:05:47,861 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:05:47,893 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:05:47,894 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:05:47,902 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:05:47,902 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694109947.8597178, 'te': 1694109947.8660073, 'initial_status': True} ====================================================================================================  [2023-09-07 18:05:47,904 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:05:47.904344 [2023-09-07 18:05:47,908 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:05:47,909 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:05:49,692 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109948, schain_finish_ts: 0 [2023-09-07 18:05:58,519 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109948, schain_finish_ts: 1693742304 [2023-09-07 18:06:03,615 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:06:03,618 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:06:03,620 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:06:03.620230 [2023-09-07 18:06:03,626 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:06:03,628 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:06:03,634 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:06:03,642 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694109963.6250944, 'te': 1694109963.6417913, 'initial_status': True} ====================================================================================================  [2023-09-07 18:06:03,644 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:06:03.644890 [2023-09-07 18:06:03,652 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:06:03,654 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:06:03,909 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109948, schain_finish_ts: 1694108304 [2023-09-07 18:06:03,965 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:06:03,966 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:06:03,968 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:06:03.968062 [2023-09-07 18:06:03,975 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:06:03,975 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:06:03,981 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:06:03,987 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694109963.9735672, 'te': 1694109963.9869463, 'initial_status': True} ====================================================================================================  [2023-09-07 18:06:03,988 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:06:03.988734 [2023-09-07 18:06:03,993 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:06:03,993 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:06:07,701 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:06:07,702 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:06:07,703 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:06:07.703830 [2023-09-07 18:06:07,712 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:06:07,713 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:06:07,721 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:06:07,727 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694109967.711036, 'te': 1694109967.7266831, 'initial_status': True} ====================================================================================================  [2023-09-07 18:06:07,730 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:06:07.730415 [2023-09-07 18:06:07,736 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:06:07,736 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:06:08,584 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109960, schain_finish_ts: 1692377364 [2023-09-07 18:06:10,466 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:06:10,466 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:06:10,467 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 18:06:12,868 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.broadcast_filter:94 - sChain 0x000002bb9043962a56bacebbde03191f32a67a4b888426ac09f83548ae92e235: Parsing broadcast events from 9654072 block to 9654093 block [2023-09-07 18:06:18,781 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109960, schain_finish_ts: 1692637668 [2023-09-07 18:06:23,178 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109960, schain_finish_ts: 1693412520 [2023-09-07 18:06:27,719 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109960, schain_finish_ts: 1693742004 [2023-09-07 18:06:29,769 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:06:29,770 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:06:29,770 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:06:29,771 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:06:29,771 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:06:29,771 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:06:29,773 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:06:29,774 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 18:06:29,804 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:06:30,054 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:06:30,183 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:06:30,291 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:06:30,292 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2552 [2023-09-07 18:06:30,293 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:06:30,293 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:06:30,294 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:06:30,294 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:06:30,295 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:06:30,301 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:06:30,304 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:06:30.304165 [2023-09-07 18:06:30,483 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:06:30,520 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:06:30,553 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:06:30,555 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:06:30,561 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:06:30,562 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694109990.5140514, 'te': 1694109990.5290537, 'initial_status': True} ====================================================================================================  [2023-09-07 18:06:30,564 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:06:30.564273 [2023-09-07 18:06:30,568 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:06:30,569 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:06:35,663 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:06:35,664 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:06:35,665 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:06:35,666 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:06:35,667 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:06:35,667 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 18:06:35,697 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:06:35,976 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:06:36,067 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:06:36,067 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:06:36,068 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 18:06:36,104 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:06:36,233 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:06:36,235 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2573 [2023-09-07 18:06:36,235 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:06:36,235 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:06:36,236 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:06:36,236 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:06:36,236 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:06:36,243 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:06:36,245 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:06:36.245186 [2023-09-07 18:06:36,395 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:06:36,433 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:06:36,461 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:06:36,463 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:06:36,470 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:06:36,470 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694109996.4317348, 'te': 1694109996.4372764, 'initial_status': True} ====================================================================================================  [2023-09-07 18:06:36,472 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:06:36.472144 [2023-09-07 18:06:36,476 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:06:36,477 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:06:43,799 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:06:43,799 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:06:43,800 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:06:43,800 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:06:43,800 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:06:43,801 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:06:43,803 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:06:43,806 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 18:06:43,832 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:06:44,006 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:06:44,007 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:06:44,007 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:06:44,007 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:06:44,008 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:06:44,008 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 18:06:44,032 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:06:44,049 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:06:44,199 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:06:44,301 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:06:44,302 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2569 [2023-09-07 18:06:44,302 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:06:44,303 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:06:44,303 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:06:44,303 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:06:44,304 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:06:44,309 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:06:44,311 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:06:44.311348 [2023-09-07 18:06:44,393 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:06:44,533 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:06:44,560 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:06:44,596 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:06:44,599 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:06:44,609 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:06:44,610 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694110004.5584397, 'te': 1694110004.5645826, 'initial_status': True} ====================================================================================================  [2023-09-07 18:06:44,612 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:06:44.612081 [2023-09-07 18:06:44,616 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:06:44,616 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:06:44,756 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694109996, schain_finish_ts: 0 [2023-09-07 18:06:44,855 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:06:44,945 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 35 -> 16 [2023-09-07 18:06:44,945 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:06:44,946 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:06:44,946 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:06:44,947 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:06:44,947 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:06:44,948 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:06:44,956 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:06:44,960 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 18:06:44,965 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:06:44,973 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2565 [2023-09-07 18:06:44,979 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:06:44,979 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:06:44,980 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:06:44,980 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:06:44,981 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:06:44,993 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:06:44,996 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:06:44.996707 [2023-09-07 18:06:44,998 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:06:45,259 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:06:45,295 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:06:45,345 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:06:45,347 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:06:45,353 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:06:45,354 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694110005.2923338, 'te': 1694110005.304997, 'initial_status': True} ====================================================================================================  [2023-09-07 18:06:45,355 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:06:45.355941 [2023-09-07 18:06:45,361 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:06:45,362 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:06:45,399 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:06:45,537 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:06:45,645 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:06:45,646 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2567 [2023-09-07 18:06:45,647 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:06:45,647 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:06:45,647 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:06:45,647 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:06:45,648 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:06:45,653 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:06:45,655 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:06:45.655411 [2023-09-07 18:06:45,795 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:06:45,827 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:06:45,866 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:06:45,869 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:06:45,879 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:06:45,880 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694110005.82523, 'te': 1694110005.8354015, 'initial_status': True} ====================================================================================================  [2023-09-07 18:06:45,884 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:06:45.884131 [2023-09-07 18:06:45,889 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:06:45,890 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:06:53,185 INFO][1][MainProcess][MainThread] - core.schains.process_manager:57 - Process manager started [2023-09-07 18:06:54,061 INFO][1][MainProcess][MainThread] - core.schains.notifications:35 - Node account has 407246400193229545 WEI [2023-09-07 18:06:54,062 INFO][1][MainProcess][MainThread] - tools.notifications.messages:188 - Saving new balance state 3020 1 [2023-09-07 18:06:54,063 INFO][1][MainProcess][MainThread] - core.schains.process_manager:95 - Fetching schains to monitor... [2023-09-07 18:06:57,853 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:06:57,853 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:06:57,855 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:06:57.855831 [2023-09-07 18:06:57,862 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:06:57,862 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:06:57,871 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:06:57,875 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694110017.8607411, 'te': 1694110017.8752744, 'initial_status': True} ====================================================================================================  [2023-09-07 18:06:57,877 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:06:57.877177 [2023-09-07 18:06:57,882 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:06:57,882 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:06:59,450 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:06:59,451 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:06:59,451 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 18:07:15,401 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:07:15,401 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:07:15,403 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:07:15.403395 [2023-09-07 18:07:15,411 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:07:15,412 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:07:15,417 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:07:15,422 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694110035.4103231, 'te': 1694110035.421862, 'initial_status': True} ====================================================================================================  [2023-09-07 18:07:15,423 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:07:15.423884 [2023-09-07 18:07:15,428 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:07:15,428 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:07:22,693 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110032, schain_finish_ts: 1693742304 [2023-09-07 18:07:24,541 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:07:24,541 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:07:24,560 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:07:24.560737 [2023-09-07 18:07:24,578 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:07:24,578 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:07:24,583 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:07:24,588 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694110044.5766633, 'te': 1694110044.5880454, 'initial_status': True} ====================================================================================================  [2023-09-07 18:07:24,591 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:07:24.591927 [2023-09-07 18:07:24,597 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:07:24,598 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:07:25,742 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110044, schain_finish_ts: 1692377364 [2023-09-07 18:07:36,862 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110044, schain_finish_ts: 1692637668 [2023-09-07 18:07:49,101 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110068, schain_finish_ts: 1693412520 [2023-09-07 18:07:51,534 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110056, schain_finish_ts: 1694108304 [2023-09-07 18:07:51,535 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110056, schain_finish_ts: 1693742004 [2023-09-07 18:07:56,093 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:07:56,093 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:07:56,094 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:07:56,094 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:07:56,095 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:07:56,095 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 18:07:56,132 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:07:56,420 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:07:56,660 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:07:56,802 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:07:56,804 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2574 [2023-09-07 18:07:56,805 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:07:56,805 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:07:56,805 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:07:56,806 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:07:56,806 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:07:56,815 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:07:56,819 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:07:56.819385 [2023-09-07 18:07:57,009 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:07:57,045 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:07:57,085 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:07:57,089 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:07:57,102 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:07:57,103 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694110077.043138, 'te': 1694110077.050908, 'initial_status': True} ====================================================================================================  [2023-09-07 18:07:57,106 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:07:57.106931 [2023-09-07 18:07:57,112 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:07:57,113 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:08:00,507 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:08:00,507 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:08:00,507 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:08:00,507 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:08:00,508 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:08:00,508 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:08:00,508 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:08:00,510 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 18:08:00,535 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:08:00,723 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:08:00,869 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:08:00,982 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:08:00,983 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2553 [2023-09-07 18:08:00,984 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:08:00,984 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:08:00,985 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:08:00,985 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:08:00,985 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:08:00,992 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:08:00,994 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:08:00.994672 [2023-09-07 18:08:01,285 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:08:01,316 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:08:01,349 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:08:01,351 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:08:01,358 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:08:01,359 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694110081.3150256, 'te': 1694110081.3211644, 'initial_status': True} ====================================================================================================  [2023-09-07 18:08:01,360 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:08:01.360889 [2023-09-07 18:08:01,365 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:08:01,366 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:08:02,035 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 55 -> 63 [2023-09-07 18:08:13,851 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:08:13,851 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:08:13,852 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:08:13,852 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:08:13,852 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:08:13,853 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:08:13,860 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:08:13,862 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 18:08:13,895 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:08:14,334 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:08:14,474 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:08:14,632 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:08:14,634 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2568 [2023-09-07 18:08:14,635 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:08:14,636 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:08:14,636 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:08:14,637 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:08:14,637 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:08:14,644 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:08:14,646 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:08:14.646920 [2023-09-07 18:08:14,804 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:08:14,831 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:08:14,864 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:08:14,866 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:08:14,873 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:08:14,874 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694110094.8290837, 'te': 1694110094.8361008, 'initial_status': True} ====================================================================================================  [2023-09-07 18:08:14,875 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:08:14.875538 [2023-09-07 18:08:14,879 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:08:14,880 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:08:16,706 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110080, schain_finish_ts: 0 [2023-09-07 18:08:21,329 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:08:21,329 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:08:21,330 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 18:08:26,554 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:08:26,555 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:08:26,555 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:08:26,555 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:08:26,556 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:08:26,556 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 18:08:26,578 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:08:26,819 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:08:26,819 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:08:26,820 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:08:26,820 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:08:26,820 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:08:26,821 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:08:26,821 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:08:26,821 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 18:08:26,845 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:08:26,896 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:08:27,124 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:08:27,337 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:08:27,479 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:08:27,480 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2566 [2023-09-07 18:08:27,481 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:08:27,481 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:08:27,481 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:08:27,482 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:08:27,482 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:08:27,488 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:08:27,490 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:08:27.490930 [2023-09-07 18:08:27,491 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:08:27,607 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:08:27,608 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2570 [2023-09-07 18:08:27,609 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:08:27,609 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:08:27,609 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:08:27,609 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:08:27,610 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:08:27,616 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:08:27,618 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:08:27.618355 [2023-09-07 18:08:27,653 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:08:27,684 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:08:27,727 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:08:27,729 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:08:27,736 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:08:27,737 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694110107.6818645, 'te': 1694110107.690762, 'initial_status': True} ====================================================================================================  [2023-09-07 18:08:27,744 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:08:27.744016 [2023-09-07 18:08:27,749 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:08:27,750 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:08:27,820 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:08:27,862 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:08:27,890 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:08:27,892 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:08:27,898 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:08:27,899 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694110107.860381, 'te': 1694110107.8674634, 'initial_status': True} ====================================================================================================  [2023-09-07 18:08:27,900 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:08:27.900829 [2023-09-07 18:08:27,907 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:08:27,907 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:08:29,944 INFO][1][MainProcess][MainThread] - core.schains.process_manager:109 - Get leaving_history for node ... [2023-09-07 18:08:33,228 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:08:33,228 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:08:33,231 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 18:08:37,007 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:08:37,007 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:08:37,009 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:08:37.009546 [2023-09-07 18:08:37,016 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:08:37,017 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:08:37,022 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:08:37,026 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694110117.0154586, 'te': 1694110117.0262423, 'initial_status': True} ====================================================================================================  [2023-09-07 18:08:37,028 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:08:37.028121 [2023-09-07 18:08:37,032 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:08:37,032 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:08:37,057 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110104, schain_finish_ts: 1693742304 [2023-09-07 18:08:38,966 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:08:38,966 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:08:38,969 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:08:38.969648 [2023-09-07 18:08:38,976 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:08:38,977 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:08:38,983 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:08:38,987 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694110118.9747722, 'te': 1694110118.987054, 'initial_status': True} ====================================================================================================  [2023-09-07 18:08:38,989 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:08:38.989474 [2023-09-07 18:08:38,993 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:08:38,994 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:08:44,909 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110104, schain_finish_ts: 1692637668 [2023-09-07 18:08:46,841 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:08:46,842 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:08:46,846 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:08:46.846026 [2023-09-07 18:08:46,855 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:08:46,855 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:08:46,863 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:08:46,867 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694110126.8529656, 'te': 1694110126.8670123, 'initial_status': True} ====================================================================================================  [2023-09-07 18:08:46,869 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:08:46.869320 [2023-09-07 18:08:46,874 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:08:46,874 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:08:49,643 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110104, schain_finish_ts: 1692377364 [2023-09-07 18:08:50,217 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:08:50,217 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:08:50,217 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:08:50,218 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:08:50,218 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:08:50,218 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 18:08:50,240 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:08:50,576 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:08:50,786 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:08:50,905 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:08:50,906 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2575 [2023-09-07 18:08:50,907 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:08:50,908 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:08:50,908 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:08:50,909 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:08:50,910 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:08:50,917 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:08:50,919 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:08:50.919976 [2023-09-07 18:08:51,067 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:08:51,094 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:08:51,125 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:08:51,127 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:08:51,138 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:08:51,139 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694110131.0927627, 'te': 1694110131.1028244, 'initial_status': True} ====================================================================================================  [2023-09-07 18:08:51,140 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:08:51.140842 [2023-09-07 18:08:51,146 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:08:51,147 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:08:53,169 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110128, schain_finish_ts: 1693412520 [2023-09-07 18:08:53,621 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110128, schain_finish_ts: 1692377616 [2023-09-07 18:09:01,904 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:09:01,904 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:09:01,905 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:09:01,905 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:09:01,905 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:09:01,906 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:09:01,908 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:09:01,910 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 18:09:01,942 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:09:02,238 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110128, schain_finish_ts: 0 [2023-09-07 18:09:02,265 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:02,504 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:02,666 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:09:02,667 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2569 [2023-09-07 18:09:02,668 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:09:02,668 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:09:02,669 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:09:02,669 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:09:02,669 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:09:02,678 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:09:02,680 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:09:02.680508 [2023-09-07 18:09:02,889 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:02,933 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:09:02,969 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:09:02,971 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:09:02,979 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:09:02,979 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694110142.930799, 'te': 1694110142.938209, 'initial_status': True} ====================================================================================================  [2023-09-07 18:09:02,982 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:09:02.982057 [2023-09-07 18:09:02,986 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:09:02,987 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:09:04,028 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110128, schain_finish_ts: 1694108304 [2023-09-07 18:09:05,254 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:09:05,254 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:09:05,255 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:09:05,255 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:09:05,255 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:09:05,255 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:09:05,260 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:09:05,261 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 18:09:05,288 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:09:05,475 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:05,620 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:05,745 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:09:05,746 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2554 [2023-09-07 18:09:05,747 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:09:05,748 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:09:05,748 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:09:05,748 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:09:05,749 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:09:05,757 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:09:05,760 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:09:05.760325 [2023-09-07 18:09:06,029 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:06,060 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:09:06,091 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:09:06,093 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:09:06,101 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:09:06,102 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694110146.0563471, 'te': 1694110146.0658183, 'initial_status': True} ====================================================================================================  [2023-09-07 18:09:06,105 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:09:06.105205 [2023-09-07 18:09:06,110 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:09:06,111 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:09:11,213 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:09:11,213 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:09:11,214 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:09:11,214 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:09:11,215 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:09:11,216 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:09:11,217 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:09:11,218 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 18:09:11,264 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:09:11,461 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:11,605 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:11,734 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:09:11,736 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2571 [2023-09-07 18:09:11,737 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:09:11,737 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:09:11,737 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:09:11,738 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:09:11,739 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:09:11,747 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:09:11,750 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:09:11.750599 [2023-09-07 18:09:11,942 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:11,977 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:09:12,013 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:09:12,015 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:09:12,024 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:09:12,024 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694110151.9755166, 'te': 1694110151.9822474, 'initial_status': True} ====================================================================================================  [2023-09-07 18:09:12,026 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:09:12.026239 [2023-09-07 18:09:12,030 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:09:12,031 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:09:13,844 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:09:13,844 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:09:13,847 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 18:09:13,850 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110128, schain_finish_ts: 0 [2023-09-07 18:09:13,936 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:09:13,936 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:09:13,942 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 18:09:17,721 INFO][1][MainProcess][MainThread] - core.schains.process_manager:117 - Got leaving sChains for the node: [] [2023-09-07 18:09:17,722 INFO][1][MainProcess][MainThread] - core.schains.process_manager:101 -  ==================================================================================================== Monitoring sChains Node ID: 16 sChains on node: [{'name': 'portly-passionate-sirius', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666281384, 'startBlock': 7803380, 'deposit': 0, 'index': 1, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x0000017723ca2', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'curly-red-alterf', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1676058936, 'startBlock': 8469646, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000001fe6cdfa', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'howling-hoedus', 'mainnetOwner': '0x1026467C91fdA40aCc7e6935d1E52f67274FD669', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666181400, 'startBlock': 7796660, 'deposit': 0, 'index': 0, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0xe5a600f4c2608', 'multitransactionMode': False, 'thresholdEncryption': False, 'active': True}, {'name': 'green-giddy-denebola', 'mainnetOwner': '0x8e835d5d9E28ffD259E8Fa8988Cacfc333A582B4', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674565164, 'startBlock': 8368049, 'deposit': 0, 'index': 4, 'generation': 1, 'originator': '0xd2001DAb6898127Be2F167B548691C87251D13C3', 'chainId': '0x00000585eb4b1', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'wan-tasty-kitalpha', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674568392, 'startBlock': 8368271, 'deposit': 0, 'index': 5, 'generation': 1, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x00000152e7be9', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'free-basic-rosy-SKALE', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1693996920, 'startBlock': 9646443, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000002bb90439', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'submissive-well-off-rotanev', 'mainnetOwner': '0x487360568fE5Ee4AE5c75782D5F63e9935B0E512', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1692361104, 'startBlock': 9539053, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000003791fa10', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}] Number of sChains on node: 7 Empty sChain structs: 0 ====================================================================================================  [2023-09-07 18:09:18,580 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:09:18,581 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain portly-passionate-sirius - Process is running: PID = 7 [2023-09-07 18:09:20,721 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:09:20,721 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:09:20,723 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:09:20.723289 [2023-09-07 18:09:20,733 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:09:20,734 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:09:20,742 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:09:20,747 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694110160.7299335, 'te': 1694110160.7467787, 'initial_status': True} ====================================================================================================  [2023-09-07 18:09:20,748 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:09:20.748564 [2023-09-07 18:09:20,753 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:09:20,753 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:09:20,810 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110128, schain_finish_ts: 1693742004 [2023-09-07 18:09:20,864 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:09:20,865 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:09:20,867 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:09:20.867667 [2023-09-07 18:09:20,875 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:09:20,875 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:09:20,881 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:09:20,885 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694110160.8729815, 'te': 1694110160.8856397, 'initial_status': True} ====================================================================================================  [2023-09-07 18:09:20,888 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:09:20.888173 [2023-09-07 18:09:20,892 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:09:20,892 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:09:21,208 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:09:21,208 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:09:21,210 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:09:21.210722 [2023-09-07 18:09:21,218 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:09:21,218 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:09:21,225 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:09:21,229 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694110161.2167983, 'te': 1694110161.229326, 'initial_status': True} ====================================================================================================  [2023-09-07 18:09:21,231 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:09:21.231267 [2023-09-07 18:09:21,236 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:09:21,237 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:09:21,252 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:09:21,253 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain curly-red-alterf - Process is running: PID = 8 [2023-09-07 18:09:22,768 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:09:22,768 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain howling-hoedus - Process is running: PID = 9 [2023-09-07 18:09:23,640 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:09:23,640 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain green-giddy-denebola - Process is running: PID = 10 [2023-09-07 18:09:24,311 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 14 -> 11 [2023-09-07 18:09:24,534 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:09:24,534 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain wan-tasty-kitalpha - Process is running: PID = 11 [2023-09-07 18:09:25,921 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:09:25,921 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:09:25,921 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:09:25,922 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:09:25,922 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:09:25,922 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 18:09:25,943 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:09:26,153 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:26,417 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:26,556 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:09:26,558 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2567 [2023-09-07 18:09:26,562 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:09:26,562 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:09:26,562 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:09:26,562 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:09:26,563 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:09:26,573 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:09:26,577 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:09:26.577039 [2023-09-07 18:09:26,746 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:26,776 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:09:26,813 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:09:26,815 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:09:26,826 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:09:26,826 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694110166.7744918, 'te': 1694110166.7825034, 'initial_status': True} ====================================================================================================  [2023-09-07 18:09:26,828 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:09:26.828732 [2023-09-07 18:09:26,833 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:09:26,833 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:09:27,975 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 18:09:27,975 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 18:09:28,733 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:09:28,733 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain free-basic-rosy-SKALE - Process is running: PID = 30622 [2023-09-07 18:09:32,521 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 18:09:32,521 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 18:09:33,410 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:09:33,410 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain submissive-well-off-rotanev - Process is running: PID = 24912 [2023-09-07 18:09:33,410 INFO][1][MainProcess][MainThread] - core.schains.process_manager:88 - Process manager procedure finished [2023-09-07 18:09:33,411 INFO][1][MainProcess][MainThread] - __main__:65 - Sleeping for 90s after run_process_manager [2023-09-07 18:09:37,798 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110164, schain_finish_ts: 1693742304 [2023-09-07 18:09:37,851 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 10 [2023-09-07 18:09:37,852 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '01408908571de25b112584fba6092c0191b5943f3b0bcd07bd273426704b7ec61226c4311ba38146a55624f9a858ebddf2da2ebce0f31e902fe66dcfc821125e2fbede706deed01c3762b4e3863d28580d253e2a3657f151e1acfdb415f1b0712d99215435e5eea347e9bca2f9e67329478ab615a501cb0028d3ec36d0b9324806a8e2cb9e1b36ceb094fd28fcdcf00589d6eae2388a69622e5661a7af5e79df2eef9c84433559abf0801178ccc006f6407a78409e18f6eabe0dac22985c493908db4625c11d03845a8d4a8c7efef58acee867cefc24637cd7a1dc8468a0fe5b115712ff0952d63f3942e553ae5b492e81b0551b4a62c292f320d057b8a8982d190b052e6b5713a8bc7e5f40f78b35c8b272cdda45f0b0b9f02143af8227010d1f02f1020febf140918781f7f6a93e1bf44ac42fbc7594b2949290c758887ffb124138d1155fb8ec08c0b83a158ce34f6bdc34493a1551ce1bb7d99c50cd3ba61d914e77b568c2ddd47384b1fa5700041aa8a82cbb3deb7f47a73b376565277c19df23d4f871d4b110dbfb27329ae70d49db4151bc4033a080fcf486faff227c07cf8a616c28d25edbb6cbdfa0c237847a5a0c2895e90554a3d3c5c18f77d8172ee981c227bad011d61f8ee20ca489ef08e3ddaf67a981ac11162c4d184f9fb903d09d01fe49156a3a33d86994dab086badeacaa889ea948a669bc737f648a961520c4e3992f54cfc05a7992f20e39e2e8d38d3f6bdd2e8e0e360788ca582982133bb3bcb7f4cf675e429d2227634074880c9c34ae6ef2db5380c52fdbff173613ac8fd52d738ac68852d56d264f80a903e7543d60fcddf0741fca8ca6d9fcda05afbb2acede4097845e32fdb04b320b7da3a8cdcd632470282883e303bae96d1cf09ab1437b26cac6bbf152a864b91981591b31b988f039538357c3404fab441ac89c39cf0bf432f10271f1fb20576c5b42531a84121aa50f85b535d3f0deff024e285d5b67f69d55db1a7a4fb9d3be4b459598bbcc262b4d37c9722890e7410bab52e04abed11bfd8c3357f454612fdc4a0de57f86ebcef6f0fe2a44aa57321d93d217eadd2bc0299f3cbaeb081a0caf1f3c0b58002dd19482c2c2f6d5694b0072a24cfcfeab7a74bbd1ef620ca161a2c4aa21ae4c3b30792c5cf16418cd260880ca15f77966bbc5bdff62d752223328a69b97b5b257e6bdcd0419fb012a23081400b2282ba552de89131b1813779692f8933b33ce975fe34f7ed45554ec3e259be3233eb0e2daabcd94718aec79786b226809d413ffc4f9e13c1cd1225b900a4e2bb0a604e9067dfb6ce0373d6d5cdb63426c1e60a44f5555989670dea7472dfb6954ca1b53ecc4f519f4c4ef1da0dc803f70797ff5374b19881fdd76a8d410abf59f9c38b06c7581fd82b6ed599266f08825956d720d5d3a27e45a30d5fd2b0ca831244288d041be97f2390bdd0618275ee015ade35f6f97272e29762fdd04245d864e5e44a67004a81a17a13e62986e6813b3b341c9cbeba0f35ff5a45916ede1af18a81a5e3b49483a5b337a5567b8a840a269e07d397f50eef330a5731858389eed59499a6e1b1ec3ece59afff7d82a8acc9655daac5c82e4a32a4806115952472ceb1831e451c097c4ba20184c3855a85f4264ae108429a175980b8c0c5729c64eb732d5fdae0014d9ee8e117df69d9497816bf5aad0cd36fbddab3a0830ea2efaa52a3f8fa0a6d141388fda52d5457ff5af03c8f70e6071018af3202cdc101be9ff9bc566fce5d746a299ab9c916cef19689f49367e530bcc5869ad1aeb6734d00b2f10a5803b152f31de3a6aa67f9af49d95d06ce33768f8d993e00cdd4d62596ccdea81612e46ea071a6946d2d41803f72d2bfcf20c57b3eaf0b4269dfcf0360d5c39347fd318b85011604355ef6c27d0b4d9c3b0f3e991b1d36203b5ee7f2f86c8641881751485f4ceceec79bd8e0ef6be16732d886d5e5875a1', 'ethKeyName': '[SGX_KEY]', 'secretShare': 'b58e0a8ed679ca4516031ff90447e3e639a6e49d4a6ca342cb2afc735dbef1ab626dc682b4b58668de23925f911de6abee7bd7c3bff44736c470876f01ed54f1b5f246695bc75b72940aa0c9e9f70625fab91352f0a2aafef74bffd401020532', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 18:09:37,857 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110176, schain_finish_ts: 1692377364 [2023-09-07 18:09:37,898 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110164, schain_finish_ts: 0 [2023-09-07 18:09:37,918 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110176, schain_finish_ts: 1693412520 [2023-09-07 18:09:38,139 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 18:09:38,139 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 10 was received and verified [2023-09-07 18:09:38,139 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 10 [2023-09-07 18:09:38,139 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 0 [2023-09-07 18:09:38,140 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '06f940322af0e17dfc73a4d839d11d5c89f2f7cebd32a6d3f821fae9ee39b69d122475e87c4caa82d10a44b74c6394f1c087915ef78c63de8b961c475bcabd521b41668ec7b10d510d7548c2cec4d48383c73b5531640df72b07c06a272cb0e921546df2bec89a021bca1a9fae56a0239179a7c19453a800503a669064fb4e7228638743f93615ba3bd8970646910095bf604e23a56c3734a1016d43e937e4d019a01ea6fbb76d7a5ff29c65c0b89642a0f7889e64e61de230479cee5565ab75153de9011900f07d55c11996aca945ff0b71fb646a6d439bc6942b36ba98d31b21353291b7e3f44638f8cefc5026bdefcb4efa5d12d6600881628fc28908098d2ff4f3671fd3a9c5eb735c55235a19b64d811f0ac79f38a2a468e99a07469d652d3b0ef8af98cb11289a2b0bd15e98b151403bda6a614b36f0dd0f4468f16ccb1b322a903ae57d2dcce8d52f5f650d9a8a95ce58effaaef3e5c86110e97e2b6527bba0ca5dc98fb78ddb900e0c04b5497f348cc3540398f296857ac2fb62574f1ce2fc82fed4cb5722f2e2fa5c4a3a45a9c98951ed73b98cf8cd4eb52d6f00a30e56798f9c32d99d29b78559089c68021e90073f95d7ccdf1ca3fa1ff74d524008651f040e65515bb2f43368bda4d7361fd07de9cfec7761f53fe133f5c0b4ba2e1d2b5b06e025e488aa6237b3e9ef06635b766566146fc259b543c0fcd376a10d35a89fc946589a04e3e4e176b8f7401c44b304c2820eec32e7b9182c2c4b301b3884bb287e5fd6b60fe2d1734d307b93a9fc68047226789b1447799ab030ea16c955c0a9427e6bc46a27007ff123b2157818cc975c6cae210752d0e32d128229260d0eb02f9d83560928ea79b6c91081ed8e9e95f2270ec69f50a30084c9e1245c889fc10833d1931dab324b51d1bef7a835521e8d549ee70c92beba9770ab01c3f6a975af0a0d0fc7ada73f2ba4793a99aafd433226ab1e2165e42f3740442b1a5cd7c2e8e91c17b758ea6e8d2feaac2c42c5438c4e5208b6a8a4f77aede304a5d8c57a527343a425115a547e7d506f5a013cdc1046de1d3cac12bf6cd15d18be1c8338e029e1c5a34b071f51fbf023e9af8b098cf790d2ebb55ea9f063cf282d4d2e15594bc4bc8d726b0da761034a4dda55f517e9d6bd8ad1d2b48466a72a6345da244fe023a8f8c3a4e5c53219928b935abbe7045f38222adef4bf2f7a0071355ed76f77c222b774d1bbe24323d71240aa3e135d6e5e56d144c6b705f807ca4fe1c223ae4c8d02384cb70dd5467ee4f18d5dd3cf0155216036a3d4d8d31e4a8ec7aa310444ea8503b166028f836c1524436a4d35f7d1d90802508bbff12d9ee452f7b086be82e2254c28e5fe76026e224713812bf42a692c1b82573e670c344a4ff8f85f6c309b895a075d29429d782160cc046d418c6fb0249e9889390ff16dd04b26f3f49d3c53d7ff5e81a35a7c04fc48dc6e30d69cbd5da25afc0f2500e62ed9032722737ef6f5c61bc715b9e21149d3804f2960c3e250f4c1cb060ab2c46a06519e18e973fd2a62b058c4b460441970a125b22b82dcff99cc25c008d6ca59f5c362903fc39c7aef7563ac127a1705db43035370b8cc3ed5947b9f14bc20b2d106ebce90966f0c338da4bf4cb09a6c60f5005c2cf60f90e82f525b280ff3bffb29b11ecaf7a63aeea9338350e6bd7b84207c3bc532732fdefc9d390300e95a4ee3360569c3a147c31490fc118391dc7ffa9b80344fd9a52b99a3c4279929e7047333205aed2a4ba8b4b564d900f90b58477c2d131a5782a6045b082f7d3fcc7512ed5e4aa4d711f8b88e11989157f280cb021b1b028cae9805279f1c2b06c627f6cbaa6efb0bd9201131f908fcd18e470c2a7ff3aea346d826dd8a24e9b5579e751b922a495ae7cb268773e54bad30e2f9b19c0c32072adc1583bb30132ae8586e423788c2128c3e774ee30bb0342c2e3e1315c4d62fcc96f94c1e', 'ethKeyName': '[SGX_KEY]', 'secretShare': 'd0a5e68a2967ce0e8a4ae7502641dcc508f0edab2535e3e8f95d580e3660524f4036b654f44e5eb0891c418ac60707b46ea7690f10dbb73e58adfdfb4863718cdb2acbecfffe31a9be792b90ad04eaa7b41176d2b2720a9872321f06b8ba6ab2', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 18:09:38,372 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 18:09:38,372 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 0 was received and verified [2023-09-07 18:09:38,372 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 0 [2023-09-07 18:09:38,372 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 1 [2023-09-07 18:09:38,373 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '165fbbf7b02bda9a8df005ac2d84332aa08bb43d47ffd79c6eb3bd705c5ec40c24e241fd220794ed87ec5f48a1c3379f235ece8ecb5cb1bbcd76d05d07da85e7051fea135cedb84e940dde6b6fc1c5476a0057023c0b10c4377948d7348e69c51e79984fc43e3a1983829e3e2c50685605a9c39b45e5c00a3794c5c337ccab3b071bfdd501f730df68da271f4274cfeac374477416f990a90b90f94518ebbb0722843f2d6377fa7fad202f1cd7b22f438a4ae96ee38c0e6f595b3be1668fe08b00d1fbd53c4ecf9abc928a5d055b70ee9b65411f19809f41e7ac4cfee8a11b872805cea45eba2735cc96522adc1c7c24efde503664a98d4083b2bf3f9a9151622e798a5f9e7e51a02e1576aa9f48b82329953f759275e9b2653c2c6d9478a39006548eb841f0fe505170b52db218194854ddecfb11c145846724699c7cbc12bb20f72b1ef7194c061e232afddc0c39cb132cbf5feb8b594de362e9d45cf668450f711a63f72f2bd3242a227e573b727742b665a4de0399460be390a1309d763606d2bca2bbd307e0213a4da1de81907c2503d8d593c6c7ffd3093b51b9d44d5e14cdf0c394059600047d8ebc84b0e30df3b0523f2c03a65f0fecd054a5fb8ce717ad56789c461ccafa58f91a4418f0455c836cf5e7151001fc313696e7aa7a22150077242d0414f5d9cb209895693129c5afd4be321f056c9ca9dd668d29c34903fcece200fb2e36aa314867be9b6024f55ce5da9cc529b875795f03dc75199c2c39d96abbfffc3e0c47ce28d4a73f3ce95a84de6a7cd48bcce1713f53da20001cfa9d2c7546cccad3f79a53bfbbc29798808856aa32a59ca14bfed9587c3c060a225b66a24e3bf84180ad7b63b13f56c3a07c887a6c0e8377714a4f00f5a0101b5f8378c1e0b664d19fb274251e8d1e798be866df7e7b80aac447637d81c73d22e4d8903a8faed7c600ca5eab09eb6ff9666ce2fecb2a752e5c7d6feeb520062e6615ce1ed93fd4930c118c164a072181648d2fc9816b8d5b04c7c0f5ecdb660bc79ec4117190545d88b4800f54e3fe6d84855d5319b717d1de888816becc012b43fedf440562cf65d19958276a4edf7a2253906324d37d5ec0dff45d511e8d13900f1e678bed3b6b74ac76c25351c6174cfd9fb60a970b23c178e71944bf0e05030e7070cf5180f29bf52b65396badc3e971604d8ca61f924d2c8223d8cf4006523f5fe0027aa6d4bac1953425e54a19a31bd0b8f8cd879ed4ca0ec3d372d903cbf2aeb95325e78fd88eb8fd156633c8d5d3d7b4d211fe4c75a3cad4d3180c20e775f5397a54c0aaa7f7b7fcf60553e5c12df8a07cb51ea06c1df50548cb1d006e03acdbb016f578ce4b16951a543030a094b65412deef72ba331d5c84bf9f1c410f01688abb2ebadf78863f22578e80fa53678109a051ff715ade6173eccc171ddf9c394aeb2e07da624ec08919f5707878f7e26b8af85d3c66558d4e6bde0999dd585895ea616539bdd91f9113f1d4dca777a8b35a379838fdeeb5ff8c3b032091233859d78247ba9c84d61c55bd823cbe60fae8bcc0d67a7f19e5949de512d6e340deab52eee7679c7370133f3e18e8b2107ba50fa63af9a06a2d189a3a149650bb126af853e0457beeefaafb9a81e23fdb08d07f808d7710ce934f28731c2a043e9c0fdb2f1cc53b5cc37de5d14c38592a960afd04998eac5b45d37168285774507c75e99867f2a6f751d059f9a6968d35c33ccf127ddf1ffcc168966d22272adbe4d82a550bfa0b50ca5b3b8da1a96f0a9c723d664ce103884401c74e27988fd9aeac932852615d053ba2ee79043683c9a157d6f7b4bd9493f01e86a22f38d89f887c8832c5e73ff4d1180048a23e98ce0258d7d1bdd31b2adc58b36605fa3fb71f4976d9a8ddd96e30cc5a339ebeefb1d8d61c24bc0d8c5902a5533019c5c6f48d0ac028b88c33ad6ebf498b505581b304c74ae7d9fcd11faab6db1d', 'ethKeyName': '[SGX_KEY]', 'secretShare': '40e3129b4afe2bcbc25a40325bba9833e51e628bf244bb0a84dd92a7d408c160f7be3b07d88ca63d543fdde39485ae1c14ea30dabd198f2fd378165f603246ab2b4c9f3ca16bb7086f34da55f9ad94c09c9e19e3039e95e0ffaf20aabcaeecb9', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 18:09:38,608 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 18:09:38,608 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 1 was received and verified [2023-09-07 18:09:38,609 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 1 [2023-09-07 18:09:38,609 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 13 [2023-09-07 18:09:38,609 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '13d23e7c9457ba132b47ce890ef6bdd9c1e5ec17e8c7781a9b340731721d35dd26b2aa030bc3a0c5b54304cc9b507c64f723518d0f35f833b94511a075d00f2c274c5cade3980160f59e55cb220c4ef737fbd647a1f980d6dea457abd0d8687f1487bd30fef8586cbde6107816c9e7df39bb18f65cac815dbc5277071dee85d5059dc7d66c7054a99a344eb07963f27142fd67874f7305cbe06d3ecad94da4713022066c0a2ffabc1beb972738439c21da22ea280503538347c317a2042e7250069046b56201d2a9b3f83e7497b4b980a6507982d24f06b45073c14a4900f9bf21941a5f6b316ba7d7cbbcd7ffe9a16b866743586d08761b996886c8b66eef3e167ac836ad4c6e5f77955e5ec5151614b8168f7b9afff089b272a3b687ef33c425a8864c8e4819d50c414be77fd81e3a08dc47c7e7e41e79c73b41c9886c702406d3505767e6bab59a0664a9b48c13030829709ff939ea71b0b8641ad447773f0d7e486b33f58673513fa289ad66e0fd58bbd735a05f47982e0518b5ef3af1bc102be34c43d7e1860faa1a575652187a6e6bdd91cd5330a0b9de543e5725ff48200abefd18b69b9929edcfe82d0790a6138c8a1a70760853f82aa53eab68beba2a42a20360636e4b38fb1146d087f60e82accdbba942279c793337ca29496be719ac2911d04951fb0be9d9f7b63c646087cec34262e0f30215a8e8cb61c9a03904842f34baec13391c35d8d5cb41624efee089dd501d316d00465ee726a4fc711d5e8f26555ff3083ccf7b351fd68e700e68e2339de2434d6460070152a568cc00302ac377148995e0da6bcd85ad00256b793ecc13c4209260b74e71809a0e92227a06ebe04774f5b114df024375c34a292800939fbb23b7452040e54dcb9dc815b74a2dd3e90119da5cc6e17488e25d192b73675e6fd66dc6d354a673e7492906d2b388394d3d9265892a44a36f1a5f6d273d655ca7712eb92b75869396f8841718892034822c65ddabc5f221492aeeec4446471057b6c67331fc607a03069702d0bf8cad2120ecb343e5453fd1c3e44ef6d7023c6825952f4f206dfa011e3905b5cd431decac51e123921dea292b8cc1da034782c9de2cadd67d8733c6b13a03ccea6afa96eef8b0f05d8d6c87735aa2a6e57df89200cb6dde47227ba229ee07f0ad0452f885e0996f5ae33c949664d17f3c1bcb68685d8897c844ecc372ad1d7d6d61437411b18a9b3ec2b7b624134005a8772d1b40a8ad7f1db0303697ec1643b00bb06fe55b2c2400f8d9bd4757b6a77d8ac311861dca11c841357cf8de1e61a22267e6dd1298219d1f9cd3ec347a13aceeefdf0f5eac3865e572e6cc0c272d4ca4919fef9c9347ef94cb87a4153724544c5e7086d02ba5a3b423aa9c521c368d2edfe602e4d1950e271ec0bc813193f84ece55f99114539bc8eeb1c9d901eeea1f7271c5abbdf79539d4151da842b6ad419f4c1f1608eeb69de0f91b5b06299b04dd32e0a4539dbc7c66c3375038da1d4f8ea3fbf9614c2e95e816b2572244116a3cdb31b6ac1ba43441f67ac639c6380addeb7af60d9afe211dded5da2ffec367b785ef7fe329180723e82fee628e85015562c3c733114ddd7f52d20d1b82103715f537f4d18a60493ba2b15b7ef166f7d3ba0d407b301fca50486c831bd3e26d3432fc01f5815182984f318ddad1627fe16aff7ea390914498c3031622cdd422ca087b284c3abd2a4948f76bb88a4cdbf5338e2de03581475f8c8b752453b1edddbcf560c1d82a9040129125313758bd877bd87efdbd80d2c3cf6ae92f434e4ea1775057ea989de749b06b77dd7deca35134e4586141abd00457c77000484007ff62d36f77ce74d05244ec75276f95ee5640b957b7922df88d5cb81b1a3927fa05a7108be978b28c7881bbe81c16162beeadd374defd52ddeb61383f24257dc685d0ea6359c370b97f57e1b0356f85643f3e09704f79b95e1119699f', 'ethKeyName': '[SGX_KEY]', 'secretShare': '4c9eb30ae6b6e51c421ca69a5bd10de76e2a36bdc927f0a2a89d44b0137aff09468e932ab75180073401daac535ab5448e7511341e6148a6c6247b5bbd641e7e1bc43013afca8349bf8395e22ee87b6df583c7eda8830b09d1cfd1b35069e602', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 18:09:38,844 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 18:09:38,844 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 13 was received and verified [2023-09-07 18:09:38,844 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 13 [2023-09-07 18:09:38,845 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 7 [2023-09-07 18:09:38,845 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '0eebd4cd6f4a0dbd0b0c99ef62cfb2eb13a3c09894039a06b5b102399091174b2a966faa66501987f95fd0e225b36bd2fddde5809f186ac7dddaf1dacef4beca09bfffa2465dacf5663fe439541e6c0a7b0c55b36e165b064f998aafa7f4292c19edd2fa45991415b6066910df4341950996ca5fbd8dead3b8a8f1e685aedeac20574af8da3e057e056e3bdbb8606c4668417d246d73eeae8bf63889e27e44d50eee11c8609917a5f6e707eddfd788a304ac2d603447a8f93a361966ad5c6a96288ce10b6800920839aa8491e2dd88901a3c6b77e1eb0532688a685bd0180dac25873b5b9d1b4c790b81a9a0737223b927f55fa9797dda49f202054dfc995d792dbb6f7eab4e8e0525a93b98d2b19ccca79b4323c79b77a2407661f10086e38119a3a9fc61f1fe2848067f1421762d170ef056c2a07a26a2cd40d8599ffb71410e5e74f6a0558f41f6eb4698b85229a90728df8130722a9507c463788e24317607647a12b502b74d686aa3ac457454e63916228a4b7daf2dc1099ff0a1550ae31b692e35bfebaddab2a547ce1f3cb30407924b9b13f64a790fd8cc7c4f73cf4c23bfecdf74fbeb3fdf00773b20a1bfe651a32dc598bef906f24ea0074375257b26e0e37813df75dd512a3cce96e40ee496026b86f20ca37ac2e1643a87b81afb0e2cdc97cb1ea6db023dd8f1cf1c84152880c807f7d65556554c52143bcce971213138767d7201ad221bfc93a44921783a78c6fa2d40e22c8da6a357ab5e552c0493c92fb665efd5ad6522db513252ae2648140686a7cc1348669450c3fe87542bf4aacf6ef1d5fd0a8960d34410f5f057ea19a88e160f84b8731e5855c011f21f11580685c5cda5b8124b295a52499c5de8365721571ba27fe152c53ac60515016ef8a4433bc2ec5bebab591f073156cdee7652c2199703e0cdb9adb4ac289c185a2d1fb9c46869f6f970257dd4ca4ade636caffcabaa88be9d26a43fc5e6db062c1c824c678795fbff0189dd7b55ef513319a7226a02bd5c7fb122b444af451c02518c3de446246cbc4921f7c1742c9075fd0e25f5ac57ef9feac69978bfa3092ddeb6533f7e946c1ef7e54eea592b2bbc1b316d6579fad0a06a98e42fb0020af05582d5c4f4cd76f9e623064f06b19deb39bc054bd68a5ec273ce2038bbcc29b0b4eeba58f5a817b6723b60f95f9942d1855ff8ae6e98a40ac6ba7988ff2e0ed39c732d6102dc59283b61e95959791254827c13a509acf32a9b4c96c62f392c3864a4d4e8eaa37023d1a377c8f10880b8d78a0fd7b6af4a678e07a835cff50a10564f553178becfcbe02d6f8f3c0007a3c3bf31e33ed4e4828ca2a70ec0321b7d473a34312929c679e945ba62b31e0580f5845e8eb30ccbdb85649fa362d22c774f20cabbf286ddd3cda83c089b96ab571f4ee048a0b83ad0b44b7db28a4500bfacae2be6d38a05fb0c690b5bfe565d1bb9c6cc79cc2b966a16a1f2a913a90ce1e74b0f59f1cf3474c9613cfeab4906f6cad2e7411b3923282dfcda972e82101697fb0a649f8571ca2c6bcb61892359758b74c0cb33f6f045e8357a7f10dc0d35128b8e47cc401708469dac81c6ef0e1cc5d1e86bd64ed8a10b7ba19c1cdb2cab634b5c9453f15e3ec714f5aa986cd5ce4ce3d22a75a5527e833802cd23b323677b466f3b89de141f98357733a0c5b1bc5940e60a0471f091e26e48215c32104c265ee8dda21fbc9e29d96877676c0dfe8c8b0944d8db6880c2351211d16902318d1fa1dda338b4aa9b2144519a4d6647cd687b57f473284f0afcfbdb09e82cd31253569aefcef6d13f64dece07fc27f4823f931d7e6d574e5329421260c204c59fb19811a4481942f5d3e1c98e81c65359c4fbe32ad4a40f2b62c14f5ca512e283a38f88117418fc632052904e4d1cd283090377ef6dc2f39f5e56eb0b8514c4b0633ef4d2e6664a66ee6bc4047c0ebc64185b8e3743e31fb9b9800d230b', 'ethKeyName': '[SGX_KEY]', 'secretShare': '72afd8ec526a3d567c6787938787d0c5e82dd8195cace0ff363750fd87e766ff969a9ef119b2ffdadfd711c5cc88bb546ab8c559a994e7e92292bf786e1df3cc7ae6e5679746db328ff5e6bb9bf5853a1dabbe325b0b79a6167fd31fa3764f6d', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 18:09:39,090 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 18:09:39,091 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 7 was received and verified [2023-09-07 18:09:39,091 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 7 [2023-09-07 18:09:39,091 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 8 [2023-09-07 18:09:39,091 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '29c5e092b10699a51e367b80a33f95c9b8f06b78086cb0162ee10b4853c4a5e813208b17f592dc2f8258e6f9c2c3d5c2aab27df96626887650729067abf4cd142721cd818abf62ba340488f1edc06c6496b69495f2301f7cf54a9f9dd918d7c915258f5eb1913ca88b4d212055e933d11f1fabe472402054bce90d3f284f0ae6026416ab8d2f15c0099d5839cc62a6bdcea672a5f9e7d76b31fe1cbf88a0ade616f2f6436e5014a3333c009d47c3001e40001da77efa9566038c8fa835aa97532a7137915f4b7d67dc4ea26f0684e735172041135fdc9f4903fe97f5e26f258c26697347ed3363743c50887778d33d8d1ae3752ae1387b445d2c1e45863e1a2d1f5f914c8b571267d1b64998ee46361d6947d86a81e3f9cc06cfbb9828fbe699243a6c9668f23ecb1f61cdfa10ff64524e76d7627bcf26178328a402b299081307a338cc9dca8beb2ca827002fab209b561780ef69eed98deb3bdaaaaeb0431117e46435c98a85ef1a1c7b6791aacb9136086502a05df23ac0bdecb553c9c26414b9da2713ef213c875c33e348ecb3cd1f3d0fcc4f4e7e4e8d88a3263d7068332c12faa1a291d139b8124f040bff3f574bfecd71b51a9c3cfff8ee3e922a43c92f20950c2c3d43d61963a7a76decc7e69351b2da45cd22392494ffc91e5dd39e164da0f77cf9ed9a6c0565df5c95cd326740c2afbfb2a8bf6ba5c7c9bea5fea720aac70ad4a7a386f72f293df2f5452652e1ff8cb8ccde04a9a70311539686b8159a279654761051d2f202da1779f5efdc8020d544340f6831087acad72e9e920341f955de1d8eaf5eb411f450a084a796927a4455ee24c893e2327c686810ff25363cc6b5fe3dec23304f84366653cf2949ebbbeac1f1af48b21b4e61e694c20e6e6c532a1caa374fc2522d6716d2e9385f9139de2fcc629b0d2d56351752a02a5908d70b583c1183646f2bf516605bee516d6e2c815709ce47f7333b1ec5ba2ac25bc32b0f531da04390d1d006ba3d083dcd8b61c62d705fb56d9cab433ad20bdd2d5c20ddc8c8e130ac154ecdce855b4b81af1805debef58c4782d68990eb28ef338bfe3fc7fb94fc8764f467acb9a1d84f06f3b81fcc6c5300e8044776eb292dcf522c5ec78282249119715fef319af26572456bf37d83565aed4109a6142d17d9c17fa574dce964fb1fc94a0571543efa49aaa4332a53f6314d37df422e03b9183b1f5efa673e200eabce9e41e1ff4f1f799de67c04e0e6f7a6508074f51e3e6c41dbf2dc4d18755b7c0188f01a4d45f0ccf764843c786ff7093b216b72192142f2f43e4aee5b9232a7efa40456e5677a8bc060a58b319a1dff35c285961dc44beca1490c678245926bd81bd008e8412305da8877368ff6fd47c614ebe01404965ab9f6faa6b5e29ad2fcfa407d08f20b4ec58e13a56ad1e36d5e44cc9102e297806f917c5be7ed6770f660580a021c902465b01832d59a2f6a8563356c15f85b1ff48d826d7b23fcdb7fc85e557c46846ce08fea95dc784adfd67ee91c176f7ecdc1f20846845fc077eb58c79b62cce4bed9eb8e70e09a425f0928f9d723cd019b01364ad6b73ea88ab7015003d12c2ce5a5556b6563b9eaaec0b23909173c3f20e929dc314c8a0f718a7a28e567b7f9d0df21dfc30cf4032334d69a4d06721a70ec12feea24af323a0b685039b01fc3a3064acf2deb096534c38c166f2b3107dac97132adff93398cfbd12427fce23f295414fd85270f8eac636314180395969c73b1a858d5a3c4cd80ad6e6434ea43a4632ea91cf1b7bbdc4733f8221cf023800619b2f4c2b8971d428d57467e41283f297f70f7964b6613a1c2d4782d80d0c9298decf2c8231cad353fe6e6c438d17f9a870ee22a83f1987ae00b7b1ed0f4ff7c57281b1aea92497486b0a14b6b2042645212d72f25a6dfa527d9721cfeb0ba00072574ebef5f61d02a372dc08f1b4205f70efa951d61152fa14dca', 'ethKeyName': '[SGX_KEY]', 'secretShare': '8392cf9d1b8a4b68597daadef5fec8672b9f41293935afeaebc609964e5f7bb83a6bb0cb18d82d9340e4ce4fd02fd3730ad5d3431959c4aff9dd31d0dbac2fbde9cb6fa5a591d22749675d9b4c1005694c9af4de3cb9e50754f4b9afa312af8c', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 18:09:39,334 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 18:09:39,335 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 8 was received and verified [2023-09-07 18:09:39,335 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 8 [2023-09-07 18:09:39,335 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 12 [2023-09-07 18:09:39,335 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '0a67b76f1b3b7c65600a00631941a08b7b00440dab457e730de3f6f14a4c0e0d302722efc91fa1a0e60518398fdd55998a9ee1cb2a6f33b20f524cf3598ba584271affe40f5a787f96bcaf7e06325ec8a0e885e36824a8ed2911f1008f2bb909286f278309da3b1d9c6fde67791066222f033d238ae4b57271f7b51c233bfe100bed67cdea18caea5642fa10f3088e8e13c9f8c87c1940b51a36fd7bd446dc742c60e6aa6937dbc0c8fc5ab90506b84aa3aaae68741e9ab1b38b6b4de9d728ce2a9fef9366460e5bd901fc79988b713c48fd54f820596fe1eb4a38ccf768c08e1cb6f7786a5e97ab0ac72166d48dcb95e23f5c06ca16c64e6350cc61bbe26e0d2eb0f19c4e9f76f962271d3347061b9cde54580ebc8200c312720fc6cb2212dc00f9a2daf4d31de7ceac0d5db3f34dd46fe7eb8da62c9c34cbefce35af218d80095abc2f1fb8306956da73e1ca77001190e8b35e764c2118c1e3f7cc63baca9a0d4e1f70ceee94a1351caec5b38a3fa788df2825a96cc3a7284d6d089fbceff81c63ff1dc2cd1271c3c9464b384c23ca5198a3f3af8817d15df6e0becabc35660db0d320ca03140dd52942add817f7394cadc0ef42b4e9d6f8b8cd1a3f1bf44d2b49265b4bfa16f0498a6b0706fc29821315bd9c3ec1b005434294884d50127c16689ef93e379d88d3e3ed7da60dfce78866f4aca701521c3c90fc106191b908227b374f04ab9508856c05cce72b55588a6fb72af69d3975a05730e60ff34076133a28e90029812b674f2d87bef31b550dc2bc4b3cd4782d242665e1580148491bd8fc9da24bb4a6b73d9650c1e152ea4278c927a808fd4d89ba337649ba0ef7187845dc5711e6a07a414ebeb6c9ae3357c65e0c7075434ad04e73fb56a961f803cc0fdd7a37194bea129beceaf2387fbff6b73265f18d056ce74dab6891dfbb01607bb158ad75b15e2c21b47726534195636f9c0bf440ecac8fd69e26becc4119cd5ba461b2ba56832fc4a29f1861f88dcad9d37870b563f4157efab3833e7c20a3390391bd90714f85931c40e154d43084f7b149ba77f0f56a8647625246a628dea94c75bf70201a05cb7e7e5a91494b00e472caf4b814e287ed479452f6b4214d6bf5c048637db39aab2c941bce485e3bad555e479e750cfb4671093a72461740af8e74d8b5f3afeb25edec0bd8c094e4b648c14326a28c2a9fc936d444c41f7a192826dbc2e3d25e829fa844390a90fca359c1b3e397f6bda2ef776576411b2884b50c807133e8f1d4d93a13b7d9409ec8c6571a58bec062ca247b11d8e618c8ab2b523487158f589202b27b8a8b044a9731c81b0f7da968b7425010173a27d0a4ec42a9da34a815beb2d4c1b0d9cef5770a50606f4e637851134afa07d9190172dcbbfbe2651fdf6e0c800d96c38cd3e1bb36235a41c651fed8baaeeb2f089f883ab5d98a831406c038c43d1f96ae04b8672ce797039072d9470f3c3efd28482f5874f44d41d4c76e20048ea0203d4874fb8d06fb434b5cf0328fcd8ea12bc63eb24590a234800af842a204a32001fc6b7323aa5a451c43b9817c47d73013677921ec761ac51627b9381e6556583ce17ba886df0f797148fa4e482c3582240d4c55a0b562827f9b1921bf8492d3043380bc0104abc3f4923213b6f77f371ff619bd3dff29e3ca449ace7243c24b34452cc65e30f8ac7b955b847cceadfc0a21be80a79b08d94c1543403e4cec0f2da1a0b9234ac738d388b152751da08f14c06f66d42c4b1d6d3912595f117399a6839dd0484040a950890645de4bd8531c9ce274922a6ed967aa7214266e40f615a4bbde95e7ac55dab11126b86b6d8503d1a836a08b7b4c8c223d0eb31123990eab9c469238c3f518075d296ea12d462e21a1eafabaf99573e3e71e05ac71cc53b4863d3ad3c5d335b3f31e964944af1724e320ece2280a00032a75703e567cab40d3109e9317c9b0831554b0b7ebb0', 'ethKeyName': '[SGX_KEY]', 'secretShare': '3893343bdf010c844c57d88684fe96b2708d3e9c16c7b56de670807816bc6fea9c8993d16dc0fab4369251ff6a7e2cf64dd57b78d861b7942b77fd1f07df82b7fa5da6f352e18df4be5fbb96e7f5c497d018605c32a14579ac8573476ee635cb', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 18:09:39,571 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 18:09:39,571 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 12 was received and verified [2023-09-07 18:09:39,572 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 12 [2023-09-07 18:09:39,572 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 6 [2023-09-07 18:09:39,572 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '1a47ba7698ccf3e206305e812589b2610f189f6973f1cfda7ed38732357862702707bfbe7f01d9640b277c0f860605523c7a00e5fcdc3cc8ffc2472d7e0a03f43048c6c74982f1474bfa8b07ec7517a103465c76e12139294cd9472a84e43ce6166066a17828b3dd75acf7b5905c9932125ea87c94ff5be8e208d061f7c655f2028232d718004549c2ef5cf75b1b30e0abbca8cdb928b649061bde38379a50df1c59e36a87b920835752a8e012a22d2b0c400086a6b075783bfd8f97798281e718ac1ce69c937e29dc82bd8ef1ea74d524849a2a1ccac5704b91d8c09186f7791bf53465286185542c3e32ee3c1c7e4507bb29d64c7a4a84faf00abbf222fd2b1f799b9ec47dc21b89a9bbdeb421c24403b134f30f958a23621dcb9a6867c21e187b1872975c4785dd98db0bf27f70e6ef58ba28499c21dc974d52b0b246d32226418fefa1a776bc4be636787c38780b21378432d130bdefb0c238bd4897f563275231de367c6c6f21ae060f82b04045470d3ba1faedb7390fef48c4a50a6f0e13a6f2d80fcc82eaf8de78014d7fdbec53f27ab2727ba143c13beb1962fc68ab20eab89e4aa0497f467eb1e91265781358168650bdad9707ef24c93c3601777c290aa7edcc7efeb4a0838a58a5b57d355ba2ae3fb8255a43de70db8e9b44b19a1113091b8ff19d63df9b2ffe7093827e70d8119917699c92de86efc49ce683500335c0cb72f3117d8de0bd1e7c552b179beba2ad8df731d9f4cc24f293748f880322579f2b23f686734ea78d15142e40e44182a5580fa60a29c8b179ca3647371e5f368382eee12d844424bcda871a1d05ce377da1b6c7a5a2d91987848ae1990aa15da40afcbbae6ab4fcf63157f322ba3520525e4106a5224235f188712f231281bdda561fc4fd23510c1aa481573fdcd3a8d5b30d77e787f5f3fdda3ace9b2ff52265e38bf7b2baf9f5ae2be9c99253c7679d05d560d0892a077e7a161e3909b37a8bf9932d69d39b8a3ae79a75b78898aefcafdd2948dd8e3cb295d28ae414794dd3002ff7f44b0440ed3b5eb7adaddba1486f23ebca858d16fdebcb408e0a900db84e6fec8bf07fb7570e62bd156ece224fe4c286531f85678bc0353b3f137af1983f3b1228104c0abef35ebdf78f2b1a3fc44a4c619ba77616476f6d082bdce296b007c8b718ea4ecd291a67abb434b6047b261f2c2e6de5242202e90f15a31abee14381509ce09d942d45831c72b5cb1e77b507f42f4425037564b6ee0cd38f6b72aae3a7c111786b0d9dba34d5fe79c1ca4a1bc3ccf4ea62dfd06af01abbd421f005f302c6841d802f684e7a3b1cf25d5e32e55b7da41dfb29e2605f0580a8186b93e96ce4f699f225d565812c878bfce9c37c6728b4f20d73ffd02e21c78ad9814bffb2c2d654ccf03bc2ffb800b294b6f90fdf46e31841a34b8b550d99c887f6ab75e6824838f731102572ce6a4922dd1d31f456c0dc54004e919c0e414d7c6e4fca7598dfdab68a74366d3db84fd4a1a2f573a7bbdace7dcadba6028da3d9f987e5b253c6c97962c72c2ef24986a0d9fb1744af9fd880c565603d1b736e8b2a914b0099fc6a89b65faa0248720c632a1ac1b92a29c81e3dc89067212f2dc28edb4b28826b8e2600ebe793ea1341338e0430343fc2d992432510d306c3e0ae54e56f8dade01bfd7bd97b1916f08e34042e03d4692d907aa61b7c9821473aafcd2c81facbd263926ae95a5509d99f85ae990b64f8f3d1c6cb60fb3b2d3576951a4365639a1887b96669fdcd5684fdfd3d56ce135618a497f536c5b42d91b57bffe12805308929f658d219629d098550f1a8349c2d915b227d131ad626d6332f060d7979561a02ca5c410e793bc0a131fcc009b23c88756191d40546255a6082d7f9910edc8a3d7e620c329b7682aa79640c5b549219f467c606a12e2b3a099e5753ffc9782ce47b6e078992304ec067b9ca3b988fb80b75a3397d5c', 'ethKeyName': '[SGX_KEY]', 'secretShare': 'b7b89f5e8cedf7ae09c58564073a779471b0089eda91a803a44779b948e35505526be4223b12d4f81f5779a49889d8eff16f6fa3dbc835946fc3578f6e31ab69c011f0e33d4a9dbed4c492d10d37919099bdb0b0e5a4849b2e0e3ed4ff9df074', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 18:09:39,816 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 18:09:39,816 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 6 was received and verified [2023-09-07 18:09:39,816 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 6 [2023-09-07 18:09:39,817 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 5 [2023-09-07 18:09:39,817 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '20f09539483b5a3cfc5c8dfe0f92d8d7311b96849d2138e55362ee8184bfdefe124fc3e445d7cc4f2bffa9c645bd0ad32e56b094fe602cfb42b93be1cf882c3404a6d9ded4578c11519eb9dc188388d35208a80884c8d7a5c1999952c9bf3892133fa2bfbaeb8b926891e176f13ccba1ee679bd7b8231f0a237e3d5741df343a15841c4ce4d081dd97a5fd79acc0c20677b1640e717b3c7ee9344f4ab4aac56500f556e6b22b10b6b490d77bb8fb35d2eded2e15b2c9eda331f6ab7c1405546c220efdb1995e8a0247fcdd6c0f8d01c34f6affed7416899d3161bd9cbfe0ff8f1a7adcc6d36c66cf468bec8491acf6f04e0d4a972d87afa49b010f063507f50d2b384eb4dce41a01825d03a58696da400610f4dc2a256b2b1e141f64296f7fa616536e3e29341545a2125e52c12722f8a73824e41746691f537e33a62dd692180dde65337985d924f0c71d3cd35d176bc3574cfa20ba06f6c60878663cbdb9bb0c8425cec63ecfb0d2f7ba49ddfb82b3c5ac63dffe91f3eaa555467144b80b460c5a6fc4cc737d9d21635fc0ffd09b4a4407e40ed8545df8d9f2c6ba9696b213242f5cc9e19bcae8369174a0a276b0c73693a8f3af66029f385b623580a4e32e155b55de42dcbac7e085dc7b631e723e3259ef39ad101a82e8221c56ec163e6d0a75436df7bde1823bfa9e31ae770df427b3c27c498ab8ec5c0ea0296351046a14e9525ee73d94eee29dd40a82e7e0ee072d8fce9eecd2ff4e7bf602cc0cd18c0ad790340fb9476e66460ed7a8b9e033152a2326b518044cb7a2b05be601b86d0ff770d6155a9659d1729503854907f5751ce44ca841d2aabbc80630100de2d92b8cce7bddb0bf81fda9065aa6b0df299a802307dd52bfccb817d15d4629b41a25994e25681e7be30c4bea2502d8c7f8be22308365f5b6bd96a4aaa1f68a5ce60c170e435caab6e8f729ee57f046c10602fb261d59422d2fc686f418a86f3ebf0d9d92a2c494b8d8f12a2511f2ac7f4fff6df467f154e29b1f5be7d94824b9de14805b99505b782588b334b027b63ea4582c8a4e866e5419f3044e3a41b09d4e22aed8fadc8acf6d3e624edaa64541b8507909e46b41f0ad187fff72ed0894b40539086c2547277360438e412b62693911bb13e7a7cd4e380b010cdff1b067bd180638ed489f380c390042424e81a5465ce05ff1c0389f1964aa5d1d130327082e1c3005e24dd9948489a1568571ce8085250f0b805570bb64f8462020bb2e0501fbf2c1165aad428dde292141cf547d799264cd04fe445aced6d2fc27c13b292fd5bf923c28341e5ae01780e3e3bd5e62335a498bf788e7f3fd5dc1024e4d651442f6e4a702beb9bad7b4549d82c990ccdf01e343c8275a41984faa6a4506d72255b5d08c41c2c8879345f58917b16ee6ab7375c77dfbe86b2a1fce2e46b2980aa79eaa5e95b121f563d97235d684707fdfbcdcd8430c992d912926cdbd89682881c480aa610769b63d6ac968937cfdb2ec975855e4856f2d58dd50c738aa892f4b5e4319b25e1c5205788f1086b014a7f09c6b115289dc80c3266698f699a903c807ebb63dde62d2367f9b7e03c258df599c7443c22ad62e3c34f12bb5a5b420e02f1def376574f24f44292d91f760dbfb1e87213f831ba9740b3bb47ad14e027168df5194bfd1582d7644fde08e34615ed3925dda568e36d53d844e8786e40b079cb270bf985211a1e5f082d717fa5bcc50959759f5044a4a9c66529809ff14936789ad99ea0746c24aaf118e31f8be9ef4ddfe6703ebe21776d4305df459265f0219750f759ba0ff996394d03366a5b0d42497987ded00e4a31dc748a7cb0ab29673b09db56251f7cc1b2259b034daff14c95b4df5bf97a06b96cce11f81280f61d7ba038b556494b0a4e2814129a99aa98691e8e07cda82c0fc6b2d36b20104301f4fb46cb0a08cdb171fcd2fc0c2fe5caf0c07597aed50001b99b36817', 'ethKeyName': '[SGX_KEY]', 'secretShare': 'aae43ea2dff9180b132307a3f7897b514b6e1ee2c941f3d8267e3023095dacc524d66c0a10b781db24615cddb6a7189a39785627f70fca3fc9947b9a1f8f258c1cb7946c41f99f65364f78dfac67a53cdfc9cf427e6c296e7b6c6d90881ce7b6', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 18:09:40,009 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 18:09:40,009 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 5 was received and verified [2023-09-07 18:09:40,010 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 5 [2023-09-07 18:09:40,010 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 4 [2023-09-07 18:09:40,010 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '11d3ed5719c1134a265dde10aad060dffe8d0c15cc37208f877afb27df60d63c1feb78f5dd690f58252dc4c8b36d2a3a759ddaf2ced8e00da00f775c827b94c225a0d37e56bf165cdf21cb9d4ae1c5b70658a0f62a15f891c44c458d17f151671727ca4a132435e46d551e7b4943c5b5ff59bacfc6c696a0b921ee8dbb00922e01a4e75feb7e6e421ddb0f0afcb130b3d464f04abd64e3765785c7511e0174de2a5ab7f5e763e25fabd0f8f25a588b636f51b49d0ba81fd4948ddf57bf6ffb22102d3328b031c2bc53000b85c34b1c119724ed3ef480bcfc27ab88dad6123a44156b983ec34d4715008d8e7465bfcb5c6c112a50a566fb780235c45b3985b1222cf93c17f2d8a261bf9bbba8b197f28c8fbbea4f2ac09548b128db708a426e0222dbed632f3a388fe4856680ab136be722d739a3f759438de814b66636d4862806714b6de9b3e8c4cf425b94e3535846bd887245b0344f2abb43f58841447db7122dfe1891abb1d8a67527ef67b85442870ac750e46d622fc04df90ea3af48a00b2718153cefc172c03c6a33239fd9d7fc5ca2d6217c5f05ce6a98035aeabd600fa5648a0970d264d6fa60663e1d61af31da352f512d4024fe3fadfba0536bc523f976814d5bf431707c870fa414cad0e313d89c65d37c32ae25d033a37ea77b1ba989f9dae2ab4c4ac1a7b511bd72373bd72cd642617a555c303ccea251dfc50d70f26855cc566780161e6ff7d11d9726dd441e1e614c388bbcc61cc7bc15862cdc0f4f82f1feac14ff60cad2fc1dc95d492d3d1df2153622eff3b73dc0fa5b120017e57e8992eddee17ce237ee455227319cfbf27a666671ca9a59966a29742ad883c249ba666e1da99af09c823bf064ec2bd3fc01db32d38992674494568a08fa9ef75032d0836bad934cf57884002292886dbaf77f0d8aef60cb3d7844ee008acd714d4da0a32335559be96c1626da7fa483ae815c4a98955dd1be7a5dad0d2b9bd8c55090cbb957db3d32688ff50c7c714862c2b0f392da25be78ab1f142061a072ae4805e062da8cab266d478fd56d9464a068328a4b2a02970e22858e28dbe11449b68f1c5ed78674c4981f7bc8911d52578db642d8e876694f282b302281699d08b34cc7dbedc3daef74e6ce9d43a436a41699e563cede852f8f313d03d7aa130782d4785af24dd3af44abe63575f50f5a52efa91cff0296a125fc4d1e13915a8e03c265dbd384ec211b9688a48a7c280082963750ff4220335f2227022e0792df337105312aa1b23fa5330541a37857125f32e76e050f29a66cb46c0f0ba631cd9b002b852ccb11d4ff1e65a8d88a029db88d727b32eb13dbdd34051964e121aa8bb9f565250177f59bbdeb46b674d5963109c7adf2c7546a34252b094fedbcf65b716e5183bb9281d95b6edbe2388a3400c62645b7361cca6eddd227e945d0960427b205debf5d77814dcea6bcf9e91addc8b3bd89b019f6aa007a19ca7f8f41f506b4c03ddcba669b800a753f45a1458a0a4f4f9440a5660094c62925c06f42a72f3e16beafcbfee07109d5514ac748947ade43a66c10cd749270105260b8b48d42549ceb8819d9fa206a072f3ad54ab02b257b824009374c43a212a026d6b1f9801cbcb3e60b015b1ed7faa114a92ca1d3be7076d1fe049302010efdef319de9461d8053f43284ac765506968cfe4c0ee1b430eb53131915a33a1e4aafa8b1c57126c108357eb1bc23a511497b7ad74f4b893ae14457f8854a321df37b341dbe82031eadcced3fba7f4f01c2656713d0152efea4febff23b43bf121a8e1a3f728d659f8a4e1ac2d6c2e50be55342c89fe6a13a7c9c8308bd662b2ce4a577afcfef4d3256e7c252e227bcd87bbbbdfd3ee1c772a018194e0fd104295e3af903938a07007d20b3d370c7c1af505c9ac8b6809e938e1a99d2be2b1419acd860f66631f30bcc1afbdf004e56853930384f0a0e0abb1b85d75dd72307', 'ethKeyName': '[SGX_KEY]', 'secretShare': '52a09a33fd6943bef89569d547a4bea2bfcab6c3f81cd8451fc2d16a30615bd5e608e3325e876fa92b101ded296cfb26a5b3d7588a61a76b23c1968e1553ab4bb0a208b46f02411358eaf64e961a476b3e2049272effbce765dfa0a77b65fbe5', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 18:09:40,223 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 18:09:40,223 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 4 was received and verified [2023-09-07 18:09:40,223 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 4 [2023-09-07 18:09:40,223 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 9 [2023-09-07 18:09:40,224 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '2717c430f43b6f9fa464b72dc401a7b51bdd40e70203364b71f7c19ec3fcdd6c17e1644fcd13aa74b384dfa540c9e379422abac440d6ca8c60f9fbebb2ccd63007bfd9586c011cc23c3ef374fcc2169c60f35b0f736b2acb3f324415a7c7d21c2f9ba816e66391256de44dd332c10f9ea90a42a1d32da8c440f44ffa8c8b4a470b863c4003c1bbe472bd904c9c61b97a931ce0b48bf2580263e61f4f72d731ad190731c80423f47084c7eccef0d5734bc013d95ca96af3e58d390c8c10008356042c283619702718c409ddd57c35f8943dcc39a9ffd10d4eec7899f835a7fac80a6090942467e712b71306fd2be0dcc78d1fd9c70cb11c41c664197bb4e01db8233ba01c9af2091998853cc856a9c8add178d7ceccc83967585d7f45e1f377e61798e95802b55dde8ddcb359b9e2eda3bc1ab5426a970bf328914873f822f3ec135c588e3435b4c455fb40a2839f226692f747fdde8862cc44a01b8f4d5c1e8d1e400ddf2b02f725dd6ae469a2688bcd65a5d6284c52310f4c9d32aa1c209a14257eb42fca13ead86275a38e8bb79dcc319e2bc5911e5baa5710f6660cd422992acff076f6c1e55f73e36bb7cd1cf037a992ba2c81d45d52cde4e5ed9162492c0eef50bb2a652d1726d337daacc6f199002b24a1c4a56474ae3a917a8232861c0baf70d2c487a048c42bdb9ddd04335ef80cad436db792983fb9eeb8ee88df161e7cb57cde64a227a33f4f499c0117a53c0b7213d983a293caaafd88e83fb5ad163bd992a29e5d929ff6370b3cd4a4e852ffbc5e47e4eea4fd53ae0d29f4d3432e86b7ea9fb7cddeae5c1d8a15ddad03b40fb5d484c6633aefda5999c175d13503d2455a3564d3fbc2135c90eb67b287080f80db72bc922d1ec7f1036e097eff2626b2a1aed6a1b8c4bf516077a756b14d2acaa2c60a39f2fb9aa7f45156e3bf088d59fd7edfeda07e97ba1686c5b5dfdf89cc8a37b430b8f22289ec38e6b2cd0afc4b7acd74fab5ca2cdd2ba955dbe86d9d73b07770d7aab408dea2b6c780b202a27413d739fdef6efbae5ebfeedf3460e94c9d50a71898731616a1b7f6605f18424cfcbe2633604d846e9cfba9f461db382894cac2a5c6ae02d0f6c35d01e70da25809e96f08697ca4e937d775572dfbdeace960c480a20da5df39c9fac45d2d273ef534e8e00afb6be9d783c258bac9d022818880ffacb349029faeb9dc04133415fedb26c926f47738fe73f68f7009de83c1d9b20beea2b3d8023d06911b2fce8d628c1b7c316b8f447c57975a9408861e81c2d4aaadc85cca3fa27c248e04cc4d9b73581f4e0d7391084a71c179921ebd889bdf0498b0faca661b07a2d71da9014a6d68cb1de4628c38f6d0a14e08725d0176482520500bdad97618d2b31af6bd2a9f6c0ffef7d6944ed89854015317a944da55c904165420074c337a1107ad031cd63cfe6f7d57c3ac4b7e0d4972dbe202ac657c8c76ff11ff6f812d6228ed2550063905dc955d8c4d3d5770dd9f5a4c727e8789c4070a5a4ed1bb01c6142883c26985fe440f04d6563d1e6c6ec21dd9a49772291823bf569d260e33031d0a6e276ec44172f144efda2b89bc9868c996c03a6146e482921bd3c882affd287e3ca609e8432578b38fc6a113a1d9c23ef74877ee2ff4b946fd52e60125ed0d70f6af74c3014a27c7c6e02a244cf67d1940824bcc8b0546838fbc91dfbbcb2d5f816e7e3f4c300d752e5a7f9e924ef27083b6a6593d580d4aaeb0483e185d0245edf7fef4392ed27884c18ca50378b3a608fd82e74a81288ad6b8f4a62e45231181bc2cd2bf75d90d09dbdb2e41a78e87d5e1150b50261f6011aa89405c7507fb0d3de5f87ff4cf27051e251218a15af38d3ca6529e407eb5d2c8881b201714006b76de72fcf1be894cefc62dad72b127246073ee203af6440523bb899e090b3e4e57b2cc69ba8e438482a9db55146ff5b6c7d023f36079a9b02b5aead0d8', 'ethKeyName': '[SGX_KEY]', 'secretShare': '8138363dd7e0ef27feb8fcbeadaa3f7468d831c4b4e109e16f74281e2e00da616810e318b0ca18312a69be474f8545f7293b8b791530004c5f51775f06ad3613f3eb3b0b45c810418d5e93e3b5c3ca0fa7816f9650e5501fb9a6c8a5c0d0453b', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 18:09:40,469 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 18:09:40,470 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 9 was received and verified [2023-09-07 18:09:40,470 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 9 [2023-09-07 18:09:40,471 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 15 [2023-09-07 18:09:40,471 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '15fe514c4913cc4b0a681379452c3fba8f3c2c050dd2500de5fba1d1d1fedfc226596f8d2f45a4c5b03b7a02b6f2b782dfd76b2818e49b8b68eb991e13b09d9d2250a25f009db651c16ca09d4d68cc89abba37b8a48c3f339386cf22b4559034185ccf41615cd5e5e65690109d1349bb0a0f8c3f6e63f1f0122e95ccab4207100e1413476585ad4f49e9a6255e467e7af3c4c564353ea03dc5c50a613df2d2881e9e33f6ffebac8d9288a21927e596d6851117fe486031935bdd8fef160f83862276d341bd902b6ea1cf14c27a7ea6809b95761d7611a5aeb28ed6f589803c3f20d9cb66e355a47556cff8bc66d2660fb91be06c83a82cc4aa2f37e046f35f440b747e7bb5ab65904d2001d4ac19529a0f1a9036ee9912b35b14ab174c8fc1d61cef574cb71592c436c86ed9c8c9ff5470b1a7106e6fcfc7d17c46e9d52969e921efc67d288971a827086891aa7f95a043941172d274ffa9cbd25b2b1852db8f19d0f66c3baca2c296944be6d83914388108a99b8e8cacf46f52e5219eeb9c8217129eaaf8d18808f97064698bd326c95ae2047d2d08c3d3a47ef89edc5834eb0cec3df004747a2429f9f8eed695c0dc9c1f3c3423678abcb3af2f24bffd970a143a13c3a4e6ec136557dcc5537bea1e13e1f38fc6dcfca82e1627eafe15b2b91e2046b8de305b8521b2b594dfb511ae7fe132ac114315d083d85485131e6cba1f776436d19ba48fbfe6501cd82ebb9459d04122959782938a43ee8b19ce0da82d7bb8152fb6ed16f77199dc06ec7911e7eedd4d6432f170895535d2f749f7a730364ffa693fa4fb9f7af3d463248015aca1f66c7633a6ceafdf2e4d35bf825f06bb6bfd5abded4397aaf51cfcfdc7e7d2cff1e9de4aad6d2123bcd60cf5b86a210969d7427c49cc1f9678bce747de3ef00139eb3463420e83a7f1a578520b7011aff16646252ed224b722c37e1727b17ba3a136cae10bd17c358f36085489392d741fd7d1bb2dd67c99d40dbefaca07f9a964265af573d3d7184c748712d3772256daf7536e6a90785642c542e6fcec53527f6e4f2259e844b88087ae732c832718821678318d7cfda3f66c58eb58d6bf497485e741f8665132de56a9393088287ffa1c002e3f9c3bb9c103b5253164eff206fad726d6f7fbf05b52dee143522a8e8dcbecb7d9eb933f20e326e018cb09446ebea228c3d26704aa64f8979d390d63f9b3991b322969572820a2e7cad3b62648a11441cd8342a5eaeaa83b393a274860782ba0dce8f8c0f7c8403f9e894164524c487abb1b021ad6fdb88233fe0a6fd0550a7f5a2f5dac38ce3b8d40271b8fa8ccedc190b39b3ab68fb71cb3c30cf2e36716cb45b94757676586dceb5404d4de15e0657638cd51a55e40d72b182d9e7b130f82a3260704ed2a6d0df34cf13f9c5bf6be7c863beb98f22996d0452ff6c6a364ad2051b416c7dfe908daa65404b180061ee34388209e095d495ae91aab7277d6f7525eae6644fe9dd149a46cc9f21990f4b5a5c9f679cf93bf2e742771e02373a4059f63aeaf782ad1826ec85a70e5fd532a0623c535fdd1fe44202334291ac6062f4b6ab631a86567f3938ad96c45e0a9c3010012754a840b3df20058a4ac8113ec0dec2b2e26ace27cbd3f7e2af34ff8e5bf1421fc847f77c9e52673bf0189add6674c9c8eae1ba516933ab71ff429fafc79a142c01fabfff8ce1e0383953abc4b1f1b163ac53478f0ef0775ed7c6f1419e3dfb2dc8117cada762f9f97b470e38659f70340a917553cbdb4ecf8535d5c2e69738ce67c8e65554b01138b2ebfb89f49c2347e88610e1c4f32e827afa6cf3cdf65fc48d369a1087823bb3299de55f634209daa61d6e0af3f4aa56305f92937118cc3eab69c51da8c21d93d371be769d4ff8895a4d8f6815338bb667df911b35db832e4bb090201c11e2eeb445fa065fe1f93183a0ef7d1ffb3d7f3fc19e93fc2e121eaca95ef72b9', 'ethKeyName': '[SGX_KEY]', 'secretShare': 'b10ecebe641f78186dcd7f1ac141c907a8fe67e19075642a4c3355ba590b2089ca0245b7f3eb12b77a349f6df4709aba24c5e614afc9d12bee85671063794050f018abb1a3861b0b52624489e1a7fc3ef891f979d6a3573d614e4920cb9ee7cc', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 18:09:40,712 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 18:09:40,712 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 15 was received and verified [2023-09-07 18:09:40,712 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 15 [2023-09-07 18:09:40,713 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 2 [2023-09-07 18:09:40,713 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '1f30027978341f8e1c397447d47208665ca24199f4364c22e8378f1c13769c88099bcd5a18e7729cb269f7c590d41e3600d0fc190402bc8298928bab6c0aee500e332874c426181e43db9714e090e72140d95445accf5b41f110f9eb719118851f0d267f4ba8f2d910d81f1eeb226b48f2e6142004045fc30b6963983651ba4703475aa22ebd6b6ca9f3aba83e288977bd2f73c925f6a0bab2e8d404aea9c4e9271d8fe5a9ebc375d6b974f43bad6ff923283a77605e5dc88e5b554a2f4a1e1f051c8d8813f5053bab76737835d1f20f63b3471efd693265f2702ea30efac39828a3e5b167a3d3b8c9b048c9f93e45434602bd92d0da9f0352598d491e451d0b01da171560275464a0bee6d96211423c955753a2d049845e671b5c6436cf4081185df4ee2772792491e555e056584964cf3758d027318d01a95dc96154e7aca9110b5ac02e74dfee2fd7f924776328ebef8822a877424615b31c0209dffecb4724e1c2f270c91c0d0b3240c67bbf9c009d1bebfce152063dbc2e1013b4d3959f17d271efbe637c1348f8dad5368dda11b1dad07c8348c1cccd415a39ff0ad31d2518d08b5c3c566e109ae0b6de86027020d7e754ee4e9ffebe206e3e28f9e50429669b5229f1cc9042e3d368f4ff8c9b8d4f0d070adae51c24ff9677bb7b4e4d22015079e1ba4d1de04bbc63564623ed42fc62d7a3beac760f69fd54345ebf64303f323a4fd287a2dfd6858c8c685c0103db575c73a32ec69e2d9f31a6c09cba14397179e26e3b743770a2483504a712ef55ac26c8815c46fb6c17c1dea7f24c0c2091db81ca4ac226db927bbb540b0af72bf021f32432a4568834f706f3b3811797da5be9a0d619c4d29dd295193724b6e605993e19ae8d4345e7ff41c83b2a0822da7240e1ebd5a3dbc8415e944dec5bde3236da6e790d60cbc93c9da2d8f62af6a08b1c682bcb0c2e3b3e5324e71e7978aa4de22dd24bd9c590109352683902026f0773f8e693cad2da33354223a6934a99b4cbfc89ffa3a24e2666351cdb1e2c1071ffaca9643a4111da86a7b10434c531b32851493e56423a061d17c45721a1ef183c04b597551d9a460fce77cf7d34464debe2abd91983797abbe9a92215dc35f15a48d143dd74ee322657d033c0fba58b99e908d37c426504645ff95818e5e00c4c1b273aedae625864fee0ad9b7cf4fe66214b9ee6041d70f661b05016c7070f37fec93fbf3c02184c4f5172cbe74ec3aee274b2e9d96572afd1fd550fe7d9257ab21a5e3fc0bd1c2d04908c745944fb6b9e7772e87f4a44287fd5b829662c5ac4b23c749e9fc342fd9eaca95132819337d662863eb495175bc89d9723a384f2d3852c354dfd557589fc53714622e0ea6cd88f4ed6125734b41ce2cb16a585dcba07521810a24a9a2d20fecfed927c6d0a1d0e91c5c324cb7fc7c08d1b03d209b19c539a4e723b34b927ef5da87a03c6868745263dff3a801b3a8fad2bda2111688c0d0181fd4e3e7b386c988ee176fabf42d6a92c1f587957b7a12d18a0b29860c39a6a2ab836e79cf5c54770f9c2e57c32c1fddcb8cf9c06e7389d1917d318c11581b97bbd43eab526b1d277260e31b57ce7f5cce78bf52e4263ff0ec4da50e5b21fd5b2dc994db0af78d826b0f254d78c4b33ce844f9b16adc19908d69ffead75cf3663aedbf48d71ddba753997d021f2580c0b3a7a6c6c25b4f3304fcd47786793b66a60da991fc2665a5712f63f6e7f31662678ec3960d3e9e1114d022afc2153e7945a787d68b9f14ae6cb72e505ffb1b1cfc523db6d8554311062df021db4b63431a2afc6eaf5e08a61879b3d781bc09c479f7811c1c229dd0cf4073adf7323b1328e00b0c64b56d9c96b06a11152e43a90c0e618f957b7e00bf0eae5706a73efe7fda65cde64e342c635af0762216ba8c23e5e68e09fb4791c3617d51fc205a485f5d790c6e08e07f68267485103d27dd23f16f0e673bdbb', 'ethKeyName': '[SGX_KEY]', 'secretShare': 'f40a45c366c25d183b4b0736591bdf557cddf67e5a50831226ebe1ca489e673222be2d28ea3ee4a31286742063de66e12a0e54448630cc9b97d8ee1fd10e6b83384d08678206811430c379b1196ee55d86817ac4c2c4e6b9007eebf68fcaf8f2', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 18:09:40,905 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 18:09:40,905 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 2 was received and verified [2023-09-07 18:09:40,905 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 2 [2023-09-07 18:09:40,905 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 11 [2023-09-07 18:09:40,906 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '2696ca3f016a2ed11ce0a880be7acb547ca580141dfceacc304a6913396dee4e0f57cb4ce9dc243b463c1e83bacccb144bb3011a739addff34b53a8a9a5e9ed116610fe3bb1d23d65ec0c654dc837d13ab5d014d2869c96a56bd2a5abb00867d13846b24c0c8217e05e482311b6cfc09147f4bc2b5dbf36664b84c6a4dc4117116a739bc6bb99fe42357f6a6fc4ec0b4b8b370be6deeb339ad22196a208a41b8112cf80672e798611e206978916ffea0dade0446f575d06bd49390641f0b5673268228611e1bb299a2fdd29b6ef8d60bccf52f7a8da4a22a9811b9f30919495d01571aed3f746f6ee1abebd7ee696f3adf31b6301ace9dcc89f7dbd120af6bd60a320bdd95caa169774d0e5f10b4e7248033f30353c863ca4d8da25b4f7be4de20fe36f75cb4dc151fdf0f3b71a19f5ae9c7c695d8acf47fd65b0c942e8291660d597fc38f06e9cf5d3a0898c436c59592345a622572a1d119c7f30520bf58f70264b1ce813e12bd07d503f90714d9c2971a82f04f7ba9697cad9b9d020af8fd0a4f1d9f1f3bc0f59b6dcc33ef1c0e9c3bd69df424a45eeb25b3c360f4b001602b5fd9b50260a13863fe68f5c6b621669c2ef31777c2055e1e3c0c45501c49112710b4db0c47c21efc2d64533f414e00298f9afe7d06d43103320a1387d599f40a2751e33e075fed4869a6ba6d730d355bbb6e8ccee547e5e940a45cba6e97c1017d4b00c7586764182998127d8f1f436898615a36dfea29733d4c17eca027721406afcd3a0e61e6d2105f6543022ea1f7956daf3d7daef9dadbaa1f5f4ae4531706f5ab3d50eb757a7a6fee4cecffac0b7b319eec15ce1f819dc6beaa655e6a03051a5745e54d82950fb1bf9db1a83d1b00dc46106ad1bcffc627b5e4c8978516af8caaa48733f082fc16c3f4d54fcf80f4bcf05a0012e154633052a0d0b22c0f06c8500a319e3717f03703e5a832bd89f8a407bdcdb82f3ce691fdb7b212ab0cea958ad021aead6cebcdcecd7151f6fad18f7da4306bd5f76fa1fc9316775827b0bae6a0d0cba9f7ffe269a53d5897b0a428d8de7fc02f316d50567a22466c2d6e7282cf7a41143d0326aa7210ea3ac0e8357445808e858289a22fb99e81bd0545137bf01a09a97238a5b9fc1978b0e4968f2f722a2d5cbb85969811c97ccf23abd3309d58b40120415e6cf02d1f8ee39f813d7cd31391a61643dbbf81f5560c4ce86e1d8e4d126e59300ea64e59ea42c7428b116d2faa24e7c3d43da4bde507b47dec61e7c4304c08127693cf233db466de779c8296215e1f94611f4b4da225d19006a2301642c1a55048725ba7233d74a7e56faad8b1863c438eae219c3d29629f2f237e425585b88bd743b881fb7c1006c7fc1c3312cf0532282d3634a203f13503261681be4345237f1b2f393d5ac2d2587048e9d9d2cd85cd67fb9473037d63615356845851b3548f81a0127143cbc721d7f9702c4d0bbecfab4946e31babf6c3ea40c4b14c77415228d2d7ace9505568243ed1a0572e7bff77caf6e428ea2daba22471a4cf2cac84e45f0830e0d4d6806cbbfdef7ef83df955c027862abb4685ab3c0d4a0888e932058299b572c04baf222ce567bea6816dafa5d1480f826c6afe86f3876e234488296010d982398cf58af9cf1ebae94b078dbae632091fd4ed2d4f0841d9d87823aeb781242bf8b706785d830094ebe0a39be0caaf2ef6a021a9a9feb194206079e5c895c01ca841678acc35d8aa16541b9c5fcfcb010fab5ed318bb3a7c39b14a4285d34451250d16710bd57980f8443d8e7a37200416e2e99a2136084885b86c776f70af09eae56df60833b982fed37b5c15fe22109592c8aa7044547e75d8819258a7cc256c7f114784c09802f3f6a4759af6582d6cbeb82ea1bf68f3d3f1cbd6fd265f324807d08d12c55b18609e2ed9c4c24c0121a604e864f3a9bc5047d7b416514b9cf31511c3a7e9143ac32dd05f4aec39', 'ethKeyName': '[SGX_KEY]', 'secretShare': '609b42c034fafa9ac51e67056119ae8d23bbddb82d69ef1f24f469cc73a8c19dd95caca9e6f138c6aaf10bf13d66abd3ad3f73c0c909b47fc011d343c319f2820cd8ad7b366e8740f763262a87391c606610840ed52c40eb8091f7a0443fce69', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 18:09:41,019 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:09:41,020 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:09:41,020 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:09:41,020 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:09:41,020 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:09:41,020 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:09:41,022 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:09:41,022 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 18:09:41,070 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:09:41,158 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 18:09:41,161 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 11 was received and verified [2023-09-07 18:09:41,166 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 11 [2023-09-07 18:09:41,167 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:120 - sChain free-basic-rosy-SKALE: receiving from node 3 [2023-09-07 18:09:41,172 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'dkgVerificationV2', 'params': {'publicShares': '0204f8c10ed0f364d22fe764dac79c5ae597b513de638f22179e6f1b2bcd40c1269a615fefc5dfe836e64b287d7a693fe67622f6e99c11956fd7db0cadaec0420d1511c318231c3e6f2629db68588ae4055e92559b8f59f9e33a30f1bd3782ac0ef6a4b4a0f1209b2bb4b3ad1aa55b32eb9d765bea71eb7079e49ea6dc186529138c43fb4347750ef2c399d02ddf0294014af29253a01351d80866765a8172442e8aac314b1e0e49e14fbdbf6a6acee7a0cbc0ed6b62609f742f6c873651550a1b590522388468d4eff8e9e3e061c05417d0a2669db5bdd7587e2d5d857268d00d56931374464b79cd56994fa04e59eec02910862aa0824ddf291e0c143d2f9423725b0a8236f7b0fd8a87953370aab1fb8e96bb14b475f59b1f8fc90b5711cf11af3406a91033560f6c4200442d9d9a5388520b67dd258ef7dc3c68a93dc3fa13949dbdcbe9ea0c6047959ee9b1cab432a65e82f4b8171ec21b4117d8c0c2b5066c29f787af33c9023047714c613e4545827097609b5cce35abab071193c76206ab079b748b891296faf4ea0d6de582071636284b127471493645f4037ce444039f705b2108414875ccee791499fc1a32bfa78cdd69781f975c54b63d82d2201687d21b33971f0e452038f35d02ae078f963a6d2dd4a37c3fcfd0fb1e4990d4145c344425f7444a6e2087a79dfd5064a7d87cd708493b477b36c88beb5021ab17411163952c82f56cba763885d0d6838773c256c07e3264e1c379c1e93509122853e22b3d0491779c1f5aa380145f46397962a04ba489e0a619559744a8777b23b0dcfdab88c5c4b1c01d8e918fa7792ca70fbd5778f25b6327d58ca8f1f51f2212965e886c3c9b5321fcbe747dad127d7e0f4033c7f1d1c99d01465a2e07d52281e6d9039ee28999a0fc126721740cf7a0cb34b716cd01a12803a67a9a7d7d1996bae5c4cff1c0f0dfa662d6340221a4c2d06fed1d4e4fa0d1f1f915b5ed3528f78de51ebe23f4d0f7ce51aa624d52f2589698eef65133a818dc1ffadc6e100cf1824e3033d60fdb138f9b3d7a1dc20fa55a227ee8f95a1671213f0363a0da0efd3b8e774e28208fe0057e1bfa32fd093bdfcfb3a87c54b9c82335c0d06a2c0d798dc88367f62c66c0d0a6ed717e0333fc52ab9d8b19017909a38f10403f41199e7b37b13974e977d6840108d6a54bec2629a3dc4d2a4f8724c74ba7bb795c1e6054cb69f88b4a3e750fca7d0a13b34184273c6877d8b05077d626fe38e30d162a6306825e03992fb6d73bb341f7bff8527d49b755f24cbdbba390d57f34e310b8256ecea7201f766733156a88051f8bc9688095676eaee1ec38afcaf767e42788012e262aba6ca205da6c3cc2232c32768802738e85ffbe7cd9fb4ff2f41601d8f7116a965b67220077f506a04f5d7d5a828990071a5bccff756f0d4a86a1031016be44883e0766db9a919260412c8fa6fda85590184dd6a3bb829eecb1f00d0646746ad54e2262267bc2b6dcdd1e0f8845784685995dcf2f6a8434fe6bbe18b3b80f363134496a58414323916d8b25ddcdb9bea2f90232e2a0474bc5656925ba76345180e05878e9144f61fc52a3bc1885414442fc51fb2787c6080a508914fe3f7af7a6919eef04ddb6b9c19662c9aa3124f0633080a87de57a5fb4f4500296a6f968267e3c4ddf1b3a63c04997983f0b4bfb6fb4ae02428d0e751b0dea1a3347e65ce6ba65b6302c12e1a5e4a5c8aee344adbab4f1944051f0e84b807006bef92bd85fd51a085127f4b1701e33dc1e08ad253f68399285138dafd1c0790e8185940355c0e3379997dba7b6dc01048cbfe8b1c3536ef6bea9dd3389a8f22258e6bf98ebd3d63f45d8db63ca38dfa6a0267b8bb2796907d879c4446dafbe1da75420ca9189a4f4c87fa53904eb97acdef402c3603293f963fd110e3400301e1bb7b2e33b2e894d567d9a631470e673c52db2c376f1ad44f794c3adec161e', 'ethKeyName': '[SGX_KEY]', 'secretShare': '9ddfa4b94ba159b12dd35e7f545a4a443fd418766ab26ef29dec1655a1de9fe5e9d3f19db537e31f12493717bf0649c5a55ada19d3420dc2ef2b6ccf83f559dbec8471301e057ed3b48d15934724db3a12641ad786739dac5393b5da95e9d942', 'n': 16, 't': 11, 'index': 14}} [2023-09-07 18:09:41,362 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:41,403 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'result': True, 'status': 0}} [2023-09-07 18:09:41,403 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:244 - sChain: free-basic-rosy-SKALE. All data from 3 was received and verified [2023-09-07 18:09:41,404 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 3 [2023-09-07 18:09:41,405 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:132 - sChain: free-basic-rosy-SKALE. Received by 14 from 14 [2023-09-07 18:09:41,406 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:137 - sChain free-basic-rosy-SKALE: total received 16 broadcasts from nodes [23, 61, 15, 64, 25, 29, 9, 11, 35, 57, 24, 20, 19, 66, 28, 16] [2023-09-07 18:09:41,698 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:09:41,699 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:09:41,699 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:09:41,699 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:09:41,700 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:09:41,700 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 18:09:41,704 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:41,736 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:09:41,830 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:09:41,832 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2555 [2023-09-07 18:09:41,833 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:09:41,833 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:09:41,833 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:09:41,834 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:09:41,834 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:09:41,843 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:09:41,843 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:09:41,844 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:09:41,844 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:09:41,845 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:09:41,846 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:09:41,846 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:09:41.846709 [2023-09-07 18:09:41,846 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:09:41,860 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:09:41,860 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 18:09:41,901 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:09:41,957 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:09:41,958 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:09:41,958 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 18:09:41,993 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:42,141 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:42,179 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:09:42,226 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:09:42,228 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:09:42,237 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:09:42,238 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694110182.176749, 'te': 1694110182.1916437, 'initial_status': True} ====================================================================================================  [2023-09-07 18:09:42,241 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:09:42.241349 [2023-09-07 18:09:42,252 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:09:42,253 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:09:42,474 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:42,655 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:42,811 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:09:42,813 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2576 [2023-09-07 18:09:42,813 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:09:42,813 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:09:42,814 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:09:42,814 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:09:42,814 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:09:42,821 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:42,822 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:09:42,825 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:09:42.824989 [2023-09-07 18:09:42,886 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110176, schain_finish_ts: 1692637668 [2023-09-07 18:09:42,892 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:09:42,892 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:09:42,894 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:09:42.894619 [2023-09-07 18:09:42,902 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:09:42,902 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:09:42,911 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:09:42,915 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694110182.9007702, 'te': 1694110182.9155536, 'initial_status': True} ====================================================================================================  [2023-09-07 18:09:42,917 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:09:42.917485 [2023-09-07 18:09:42,924 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:09:42,925 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:09:42,979 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:09:42,988 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2572 [2023-09-07 18:09:42,989 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:09:42,989 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:09:42,989 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:09:42,989 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:09:42,990 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:09:42,997 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:09:42,999 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:09:42.999442 [2023-09-07 18:09:43,064 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:43,098 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:09:43,142 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:09:43,145 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:09:43,155 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:09:43,155 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694110183.096559, 'te': 1694110183.1062987, 'initial_status': True} ====================================================================================================  [2023-09-07 18:09:43,158 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:09:43.158293 [2023-09-07 18:09:43,163 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:09:43,164 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:09:43,213 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:43,244 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:09:43,278 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:09:43,280 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:09:43,288 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:09:43,288 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694110183.2425618, 'te': 1694110183.248648, 'initial_status': True} ====================================================================================================  [2023-09-07 18:09:43,289 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:09:43.289852 [2023-09-07 18:09:43,295 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:09:43,296 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:09:43,698 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:09:43,699 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:09:43,701 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:09:43.701268 [2023-09-07 18:09:43,708 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:09:43,709 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:09:43,714 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:09:43,718 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694110183.7073472, 'te': 1694110183.718506, 'initial_status': True} ====================================================================================================  [2023-09-07 18:09:43,720 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:09:43.720316 [2023-09-07 18:09:43,724 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:09:43,724 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:09:43,979 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 56 -> 60 [2023-09-07 18:09:44,043 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:09:44,043 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:09:44,043 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:09:44,044 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:09:44,044 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:09:44,044 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:09:44,045 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:09:44,046 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 18:09:44,076 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:09:44,433 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:44,594 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:44,715 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:09:44,716 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2570 [2023-09-07 18:09:44,717 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:09:44,717 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:09:44,718 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:09:44,718 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:09:44,719 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:09:44,728 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:09:44,729 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:09:44.729923 [2023-09-07 18:09:44,909 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110176, schain_finish_ts: 1694108304 [2023-09-07 18:09:44,924 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:44,948 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:09:44,980 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:09:44,983 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:09:44,993 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:09:44,994 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694110184.9468265, 'te': 1694110184.9527977, 'initial_status': True} ====================================================================================================  [2023-09-07 18:09:44,995 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:09:44.995901 [2023-09-07 18:09:45,000 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:09:45,000 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:09:46,020 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:09:46,020 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:09:46,024 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:09:46.024326 [2023-09-07 18:09:46,037 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:09:46,037 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:09:46,040 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:09:46,041 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:09:46,042 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 18:09:46,046 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:09:46,052 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694110186.0327392, 'te': 1694110186.052175, 'initial_status': True} ====================================================================================================  [2023-09-07 18:09:46,055 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:09:46.055214 [2023-09-07 18:09:46,060 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:09:46,060 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:09:50,091 ERROR][24912][submissive-well-off-rotanev][T_0] - core.schains.task:26 - Task submissive-well-off-rotanev-config failed with max() arg is an empty sequence Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 63, in execute self.am.upstream_config() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 194, in upstream_config new_config = create_new_upstream_config( File "/usr/src/admin/core/schains/config/main.py", line 51, in create_new_upstream_config schain_config = generate_schain_config_with_skale( File "/usr/src/admin/core/schains/config/generator.py", line 213, in generate_schain_config_with_skale node_groups = get_previous_schain_groups(skale, schain_name) File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 56, in get_previous_schain_groups _add_previous_schain_rotations_state( File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 119, in _add_previous_schain_rotations_state new_node_id = max(previous_nodes.items(), key=lambda x: x[1]['finish_ts'])[0] ValueError: max() arg is an empty sequence [2023-09-07 18:09:50,092 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:09:50,958 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110188, schain_finish_ts: 1693742004 [2023-09-07 18:09:52,083 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:09:52,083 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:09:52,083 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:09:52,083 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:09:52,084 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:09:52,084 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 18:09:52,103 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:09:52,356 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:52,521 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:52,637 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:09:52,638 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2568 [2023-09-07 18:09:52,639 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:09:52,639 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:09:52,639 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:09:52,640 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:09:52,640 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:09:52,646 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:09:52,648 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:09:52.648543 [2023-09-07 18:09:52,808 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:09:52,837 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:09:52,875 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:09:52,877 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:09:52,885 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:09:52,886 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694110192.8356717, 'te': 1694110192.8451018, 'initial_status': True} ====================================================================================================  [2023-09-07 18:09:52,887 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:09:52.887951 [2023-09-07 18:09:52,892 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:09:52,893 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:09:58,492 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 18:09:58,661 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:09:58,661 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 18:09:58,662 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 18:09:58,662 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 18:09:58,831 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 18:09:58,832 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 18:09:58,839 INFO][8][curly-red-alterf][T_1] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 18:09:58,847 ERROR][8][curly-red-alterf][T_1] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:09:58,848 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:09:59,128 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 18:09:59,307 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:09:59,308 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:09:59,308 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:09:59,308 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 18:09:59,478 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 18:09:59,478 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 18:09:59,480 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 18:09:59,487 ERROR][11][wan-tasty-kitalpha][T_0] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:09:59,487 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:10:02,905 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110188, schain_finish_ts: 0 [2023-09-07 18:10:04,077 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:10:04,077 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:10:04,078 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:40 - Task submissive-well-off-rotanev-config finished with None [2023-09-07 18:10:04,079 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:42 - Running task submissive-well-off-rotanev-config [2023-09-07 18:10:04,079 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 18:10:05,798 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': False} [2023-09-07 18:10:05,799 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:10:05,800 INFO][24912][submissive-well-off-rotanev][T_0] - web.models.schain:152 - Changing monitor_last_seen for submissive-well-off-rotanev to 2023-09-07 18:10:05.800835 [2023-09-07 18:10:05,805 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.action:190 - Creating new upstream_config rotation_id: 8, stream: regression-2.2 [2023-09-07 18:10:05,808 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.config.main:49 - Generating sChain config for submissive-well-off-rotanev [2023-09-07 18:10:11,812 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110200, schain_finish_ts: 1693412520 [2023-09-07 18:10:12,595 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.main:68 - sChain free-basic-rosy-SKALE: No complaints sent in schain - sending alright ... [2023-09-07 18:10:12,595 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:302 - sChain free-basic-rosy-SKALE sending alright transaction [2023-09-07 18:10:12,644 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110200, schain_finish_ts: 1693742304 [2023-09-07 18:10:12,957 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:10:12,957 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:10:12,958 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:10:12,958 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:10:12,959 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:10:12,959 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:10:12,961 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:10:12,963 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 18:10:12,990 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:10:13,183 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:10:13,356 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:10:13,490 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:10:13,491 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2573 [2023-09-07 18:10:13,492 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:10:13,492 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:10:13,492 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:10:13,493 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:10:13,493 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:10:13,499 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:10:13,501 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:10:13.501669 [2023-09-07 18:10:13,651 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.transactions.tools:46 - Dry run tx: alright, sender: 0x523E13D962A7614752eFa3dB62e0466dF0d9E89D, wallet: RedisWalletAdapter, value: 0, [2023-09-07 18:10:13,772 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:10:13,772 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:10:13,773 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:10:13,773 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 18:10:13,773 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 18:10:13,773 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:10:13,777 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:10:13,780 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 18:10:13,801 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:10:13,804 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:10:13,838 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:10:13,877 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:10:13,880 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:10:13,889 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:10:13,890 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694110213.836742, 'te': 1694110213.8450854, 'initial_status': True} ====================================================================================================  [2023-09-07 18:10:13,892 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:10:13.892746 [2023-09-07 18:10:13,899 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:10:13,900 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:10:14,025 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.transactions.tools:60 - Estimated gas for alright: 1000000 [2023-09-07 18:10:14,187 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.transactions.tools:126 - Tx: alright, Fields: {'chainId': 5, 'gas': 1000000, 'nonce': 316, 'value': 0, 'gasPrice': 10000000000, 'type': 1, 'to': '0x69E2C9576B36c8d7938Ba45a1C909B67EC53048E', 'data': '0xb9799682000002bb9043962a56bacebbde03191f32a67a4b888426ac09f83548ae92e2350000000000000000000000000000000000000000000000000000000000000010'}, [2023-09-07 18:10:14,188 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.wallets.redis_wallet:135 - Sending {'chainId': 5, 'gas': 1000000, 'nonce': 316, 'value': 0, 'gasPrice': 10000000000, 'type': 1, 'to': '0x69E2C9576B36c8d7938Ba45a1C909B67EC53048E', 'data': '0xb9799682000002bb9043962a56bacebbde03191f32a67a4b888426ac09f83548ae92e2350000000000000000000000000000000000000000000000000000000000000010'} to redis pool, method: dkg.alright [2023-09-07 18:10:14,188 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.wallets.redis_wallet:145 - Adding tx b'tx-73020fa77c49c35d' to the pool [2023-09-07 18:10:14,189 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.wallets.redis_wallet:147 - Saving tx b'tx-73020fa77c49c35d' record: b'{"status": "PROPOSED", "score": 51694110214, "multiplier": 2, "tx_hash": null, "method": "dkg.alright", "meta": null, "chainId": 5, "gas": null, "nonce": 316, "value": 0, "gasPrice": 10000000000, "type": 1, "to": "0x69E2C9576B36c8d7938Ba45a1C909B67EC53048E", "data": "0xb9799682000002bb9043962a56bacebbde03191f32a67a4b888426ac09f83548ae92e2350000000000000000000000000000000000000000000000000000000000000010"}' [2023-09-07 18:10:14,209 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:10:14,434 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:10:14,602 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:10:14,603 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2577 [2023-09-07 18:10:14,604 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:10:14,604 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:10:14,604 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:10:14,604 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:10:14,605 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:10:14,612 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:10:14,615 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:10:14.615597 [2023-09-07 18:10:14,829 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:10:14,860 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:10:14,860 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:10:14,862 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:10:14,862 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:10:14.862359 [2023-09-07 18:10:14,869 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:10:14,870 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:10:14,875 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:10:14,880 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694110214.8686168, 'te': 1694110214.8803859, 'initial_status': True} ====================================================================================================  [2023-09-07 18:10:14,882 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:10:14.882172 [2023-09-07 18:10:14,889 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:10:14,890 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:10:14,921 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:10:14,923 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:10:14,933 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:10:14,934 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694110214.8583148, 'te': 1694110214.885857, 'initial_status': True} ====================================================================================================  [2023-09-07 18:10:14,938 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:10:14.938067 [2023-09-07 18:10:14,944 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:10:14,944 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:10:15,616 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 18:10:15,616 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:10:15,618 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:10:15.618840 [2023-09-07 18:10:15,626 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:10:21,625 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110212, schain_finish_ts: 1694108304 [2023-09-07 18:10:22,385 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110212, schain_finish_ts: 1692377364 [2023-09-07 18:10:22,614 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110212, schain_finish_ts: 1693742004 [2023-09-07 18:10:24,126 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:10:24,127 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:10:24,127 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 18:10:27,912 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:10:27,913 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:10:27,913 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:10:27,913 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:10:27,913 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:10:27,914 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:10:27,915 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:10:27,918 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 18:10:27,943 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:10:28,191 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:10:28,360 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:10:28,481 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:10:28,482 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2556 [2023-09-07 18:10:28,483 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:10:28,483 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:10:28,483 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:10:28,484 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:10:28,484 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:10:28,491 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:10:28,493 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:10:28.493193 [2023-09-07 18:10:28,686 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:10:28,707 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110224, schain_finish_ts: 1692637668 [2023-09-07 18:10:28,715 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:10:28,743 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:10:28,746 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:10:28,752 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:10:28,753 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694110228.7122228, 'te': 1694110228.7198908, 'initial_status': True} ====================================================================================================  [2023-09-07 18:10:28,754 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:10:28.754955 [2023-09-07 18:10:28,759 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:10:28,759 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:10:28,923 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:10:28,923 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:10:28,923 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:10:28,924 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 18:10:28,924 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 18:10:28,924 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:10:28,932 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:10:28,933 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 18:10:28,956 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:10:29,363 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:10:29,549 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:10:29,661 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:10:29,662 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2569 [2023-09-07 18:10:29,662 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:10:29,663 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:10:29,663 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:10:29,663 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:10:29,663 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:10:29,670 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:10:29,672 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:10:29.672445 [2023-09-07 18:10:29,841 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:10:29,872 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:10:29,905 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:10:29,907 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:10:29,915 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:10:29,915 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694110229.871239, 'te': 1694110229.87827, 'initial_status': True} ====================================================================================================  [2023-09-07 18:10:29,917 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:10:29.917723 [2023-09-07 18:10:29,926 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:10:29,926 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:10:30,845 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:10:30,845 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:10:30,845 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:10:30,845 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:10:30,846 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:10:30,846 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:10:30,849 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:10:30,852 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 18:10:30,874 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:10:31,091 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:10:31,250 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:10:31,392 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:10:31,393 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2571 [2023-09-07 18:10:31,394 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:10:31,394 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:10:31,394 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:10:31,395 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:10:31,396 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:10:31,407 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:10:31,415 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:10:31.415831 [2023-09-07 18:10:31,612 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:10:31,613 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:10:31,615 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:10:31.615144 [2023-09-07 18:10:31,623 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:10:31,623 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:10:31,629 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:10:31,634 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694110231.621662, 'te': 1694110231.6340034, 'initial_status': True} ====================================================================================================  [2023-09-07 18:10:31,636 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:10:31.636194 [2023-09-07 18:10:31,640 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:10:31,641 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:10:31,667 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:10:31,700 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:10:31,728 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:10:31,730 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:10:31,737 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:10:31,738 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694110231.697398, 'te': 1694110231.7051713, 'initial_status': True} ====================================================================================================  [2023-09-07 18:10:31,740 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:10:31.740940 [2023-09-07 18:10:31,745 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:10:31,746 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:10:33,940 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 18:10:33,940 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:10:33,944 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:10:33.944410 [2023-09-07 18:10:33,954 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:10:34,172 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110224, schain_finish_ts: 0 [2023-09-07 18:10:34,326 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:10:34,327 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:10:34,329 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:10:34.329525 [2023-09-07 18:10:34,339 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:10:34,339 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:10:34,347 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:10:34,351 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694110234.3380387, 'te': 1694110234.3512146, 'initial_status': True} ====================================================================================================  [2023-09-07 18:10:34,353 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:10:34.353466 [2023-09-07 18:10:34,358 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:10:34,358 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:10:35,858 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:10:35,858 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:10:35,858 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 18:10:40,996 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110224, schain_finish_ts: 1693412520 [2023-09-07 18:10:41,123 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110224, schain_finish_ts: 1693742304 [2023-09-07 18:10:46,778 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:10:46,779 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:10:46,779 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:10:46,779 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:10:46,780 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:10:46,780 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 18:10:46,813 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:10:47,082 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:10:47,268 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:10:47,387 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:10:47,388 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2578 [2023-09-07 18:10:47,389 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:10:47,389 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:10:47,389 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:10:47,390 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:10:47,390 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:10:47,397 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:10:47,401 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:10:47.401454 [2023-09-07 18:10:47,578 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:10:47,617 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:10:47,651 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:10:47,654 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:10:47,661 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:10:47,661 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694110247.614319, 'te': 1694110247.6253133, 'initial_status': True} ====================================================================================================  [2023-09-07 18:10:47,663 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:10:47.663408 [2023-09-07 18:10:47,668 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:10:47,668 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:10:53,841 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:10:53,842 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:10:53,842 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:10:53,842 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:10:53,843 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:10:53,843 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:10:53,846 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:10:53,850 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 18:10:53,883 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:10:54,115 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:10:54,307 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:10:54,416 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:10:54,418 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2574 [2023-09-07 18:10:54,419 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:10:54,419 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:10:54,419 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:10:54,419 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:10:54,420 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:10:54,430 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:10:54,432 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:10:54.432575 [2023-09-07 18:10:54,599 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:10:54,641 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:10:54,675 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:10:54,678 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:10:54,686 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:10:54,687 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694110254.6392765, 'te': 1694110254.646735, 'initial_status': True} ====================================================================================================  [2023-09-07 18:10:54,689 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:10:54.689236 [2023-09-07 18:10:54,693 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:10:54,694 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:10:57,620 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110248, schain_finish_ts: 1692637668 [2023-09-07 18:10:57,732 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110248, schain_finish_ts: 1694108304 [2023-09-07 18:11:02,106 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:11:02,106 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:11:02,106 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:11:02,107 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:11:02,107 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:11:02,107 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:11:02,110 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:11:02,112 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 18:11:02,136 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:11:02,394 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:11:02,538 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:11:02,638 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:11:02,639 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2572 [2023-09-07 18:11:02,640 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:11:02,640 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:11:02,641 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:11:02,641 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:11:02,642 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:11:02,647 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:11:02,649 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:11:02.649720 [2023-09-07 18:11:02,847 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:11:02,900 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:11:02,932 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:11:02,934 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:11:02,941 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:11:02,941 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694110262.8963492, 'te': 1694110262.9065163, 'initial_status': True} ====================================================================================================  [2023-09-07 18:11:02,943 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:11:02.943260 [2023-09-07 18:11:02,947 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:11:02,948 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:11:03,501 INFO][1][MainProcess][MainThread] - core.schains.cleaner:65 - Cleaner process started [2023-09-07 18:11:03,536 INFO][30998][cleaner][MainThread] - core.schains.cleaner:110 - Cleaner procedure started. [2023-09-07 18:11:04,639 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:11:04,640 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:11:04,641 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 18:11:04,697 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:11:04,697 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:11:04,699 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:11:04.699432 [2023-09-07 18:11:04,705 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:11:04,706 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:11:04,712 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:11:04,716 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694110264.7045975, 'te': 1694110264.7160783, 'initial_status': True} ====================================================================================================  [2023-09-07 18:11:04,717 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:11:04.717757 [2023-09-07 18:11:04,721 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:11:04,721 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:11:05,345 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110248, schain_finish_ts: 1693742004 [2023-09-07 18:11:06,095 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110248, schain_finish_ts: 1692377364 [2023-09-07 18:11:09,265 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:11:09,265 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:11:09,267 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:11:09.267932 [2023-09-07 18:11:09,275 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:11:09,275 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:11:09,281 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:11:09,286 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694110269.2734485, 'te': 1694110269.2859542, 'initial_status': True} ====================================================================================================  [2023-09-07 18:11:09,288 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:11:09.288249 [2023-09-07 18:11:09,292 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:11:09,293 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:11:09,459 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:11:09,459 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:11:09,459 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:11:09,459 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:11:09,459 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:11:09,460 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 18:11:09,485 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:11:09,717 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:11:09,847 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:11:09,952 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:11:09,953 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2570 [2023-09-07 18:11:09,954 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:11:09,954 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:11:09,955 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:11:09,955 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:11:09,955 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:11:09,961 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:11:09,963 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:11:09.963704 [2023-09-07 18:11:10,100 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:11:10,148 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:11:10,181 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:11:10,183 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:11:10,189 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:11:10,190 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694110270.1459792, 'te': 1694110270.1536403, 'initial_status': True} ====================================================================================================  [2023-09-07 18:11:10,191 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:11:10.191632 [2023-09-07 18:11:10,195 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:11:10,196 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:11:11,489 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110248, schain_finish_ts: 0 [2023-09-07 18:11:11,896 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:11:11,896 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:11:11,897 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:11:11,897 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:11:11,897 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:11:11,898 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:11:11,899 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:11:11,903 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 18:11:11,927 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:11:12,121 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:11:12,363 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:11:12,454 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:11:12,455 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2557 [2023-09-07 18:11:12,456 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:11:12,456 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:11:12,456 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:11:12,457 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:11:12,457 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:11:12,464 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:11:12,467 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:11:12.467041 [2023-09-07 18:11:12,620 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:11:12,647 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:11:12,681 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:11:12,684 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:11:12,694 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:11:12,695 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694110272.6455827, 'te': 1694110272.652157, 'initial_status': True} ====================================================================================================  [2023-09-07 18:11:12,697 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:11:12.697469 [2023-09-07 18:11:12,702 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:11:12,703 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:11:15,457 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:11:15,462 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:11:15,463 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 18:11:26,852 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:11:26,852 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:11:26,855 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:11:26.855742 [2023-09-07 18:11:26,863 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:11:26,863 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:11:26,870 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:11:26,875 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694110286.8617783, 'te': 1694110286.8753917, 'initial_status': True} ====================================================================================================  [2023-09-07 18:11:26,882 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:11:26.882023 [2023-09-07 18:11:26,888 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:11:26,889 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:11:31,315 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110284, schain_finish_ts: 1693412520 [2023-09-07 18:11:32,547 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110284, schain_finish_ts: 1694108304 [2023-09-07 18:11:32,754 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110284, schain_finish_ts: 1692637668 [2023-09-07 18:11:34,514 INFO][30998][cleaner][MainThread] - core.schains.cleaner:116 - sChains on contracts: ['portly-passionate-sirius', 'curly-red-alterf', 'howling-hoedus', 'green-giddy-denebola', 'wan-tasty-kitalpha', 'free-basic-rosy-SKALE', 'submissive-well-off-rotanev'] sChains on node: ['curly-red-alterf', 'free-basic-rosy-SKALE', 'green-giddy-denebola', 'howling-hoedus', 'portly-passionate-sirius', 'submissive-well-off-rotanev', 'wan-tasty-kitalpha'] [2023-09-07 18:11:34,514 INFO][30998][cleaner][MainThread] - core.schains.cleaner:132 - Cleanup procedure finished [2023-09-07 18:11:34,519 INFO][1][MainProcess][MainThread] - core.schains.cleaner:67 - Cleaner process is joined. [2023-09-07 18:11:34,519 INFO][1][MainProcess][MainThread] - core.schains.cleaner:68 - Terminating the process [2023-09-07 18:11:34,520 INFO][1][MainProcess][MainThread] - __main__:70 - Sleeping for 90s after run_cleaner [2023-09-07 18:11:35,402 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110284, schain_finish_ts: 1693742304 [2023-09-07 18:11:36,602 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 18:11:36,603 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 18:11:39,419 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:11:39,419 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:11:39,419 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:11:39,420 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:11:39,420 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:11:39,420 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:11:39,420 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:11:39,421 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 18:11:39,456 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:11:39,643 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:11:39,643 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:11:39,644 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 18:11:39,690 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:11:39,895 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:11:39,895 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:11:39,895 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:11:39,895 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:11:39,896 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:11:39,896 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:11:39,901 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:11:39,902 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 18:11:39,933 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:11:40,017 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:11:40,133 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:11:40,136 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2575 [2023-09-07 18:11:40,137 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:11:40,137 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:11:40,137 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:11:40,138 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:11:40,138 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:11:40,144 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:11:40,146 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:11:40.146649 [2023-09-07 18:11:40,170 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:11:40,315 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:11:40,357 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:11:40,369 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110296, schain_finish_ts: 1692377364 [2023-09-07 18:11:40,386 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:11:40,389 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:11:40,397 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:11:40,397 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694110300.3560407, 'te': 1694110300.361993, 'initial_status': True} ====================================================================================================  [2023-09-07 18:11:40,399 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:11:40.399971 [2023-09-07 18:11:40,404 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:11:40,405 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:11:40,528 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:11:40,641 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:11:40,643 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2573 [2023-09-07 18:11:40,644 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:11:40,644 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:11:40,644 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:11:40,645 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:11:40,645 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:11:40,651 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:11:40,653 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:11:40.653176 [2023-09-07 18:11:40,901 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:11:40,929 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:11:40,960 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:11:40,963 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:11:40,972 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:11:40,972 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694110300.9274192, 'te': 1694110300.9337397, 'initial_status': True} ====================================================================================================  [2023-09-07 18:11:40,974 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:11:40.974298 [2023-09-07 18:11:40,979 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:11:40,979 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:11:41,753 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:11:41,755 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:11:41,757 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:11:41,758 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:11:41,758 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:11:41,758 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 18:11:41,780 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:11:42,006 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:11:42,173 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:11:42,218 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110296, schain_finish_ts: 0 [2023-09-07 18:11:42,277 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:11:42,278 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2579 [2023-09-07 18:11:42,279 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:11:42,279 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:11:42,279 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:11:42,280 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:11:42,280 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:11:42,285 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:11:42,288 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:11:42.288137 [2023-09-07 18:11:42,422 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:11:42,451 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:11:42,480 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:11:42,483 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:11:42,489 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:11:42,490 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694110302.4494612, 'te': 1694110302.455841, 'initial_status': True} ====================================================================================================  [2023-09-07 18:11:42,491 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:11:42.491845 [2023-09-07 18:11:42,496 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:11:42,496 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:11:47,516 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:11:47,516 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:11:47,519 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:11:47.519821 [2023-09-07 18:11:47,528 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:11:47,528 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:11:47,534 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:11:47,539 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694110307.5264812, 'te': 1694110307.539485, 'initial_status': True} ====================================================================================================  [2023-09-07 18:11:47,541 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:11:47.541776 [2023-09-07 18:11:47,545 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:11:47,546 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:11:47,571 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:11:47,572 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:11:47,573 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:11:47.573970 [2023-09-07 18:11:47,580 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:11:47,580 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:11:47,586 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:11:47,590 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694110307.5789025, 'te': 1694110307.5903966, 'initial_status': True} ====================================================================================================  [2023-09-07 18:11:47,592 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:11:47.592184 [2023-09-07 18:11:47,596 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:11:47,596 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:11:47,676 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:11:47,677 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:11:47,677 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:11:47,677 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:11:47,677 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:11:47,678 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:11:47,679 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:11:47,682 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 18:11:47,708 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:11:47,954 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:11:48,205 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:11:48,347 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:11:48,348 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2558 [2023-09-07 18:11:48,348 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:11:48,349 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:11:48,349 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:11:48,349 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:11:48,350 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:11:48,358 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:11:48,360 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:11:48.360408 [2023-09-07 18:11:48,577 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:11:48,610 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:11:48,646 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:11:48,650 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:11:48,658 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:11:48,659 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694110308.6085315, 'te': 1694110308.6147652, 'initial_status': True} ====================================================================================================  [2023-09-07 18:11:48,660 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:11:48.660935 [2023-09-07 18:11:48,665 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:11:48,666 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:11:53,569 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:11:53,570 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:11:53,571 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 18:11:55,038 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110296, schain_finish_ts: 1693742004 [2023-09-07 18:11:59,577 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:11:59,577 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:11:59,579 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:11:59.579185 [2023-09-07 18:11:59,585 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:11:59,585 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:11:59,591 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:11:59,595 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694110319.5843487, 'te': 1694110319.5951498, 'initial_status': True} ====================================================================================================  [2023-09-07 18:11:59,596 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:11:59.596977 [2023-09-07 18:11:59,601 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:11:59,601 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:12:05,156 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110320, schain_finish_ts: 1694108304 [2023-09-07 18:12:05,899 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:12:05,899 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:12:05,899 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:12:05,900 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:12:05,900 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:12:05,900 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 26 [2023-09-07 18:12:05,930 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:12:05,958 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110320, schain_finish_ts: 1693412520 [2023-09-07 18:12:06,277 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:06,430 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:06,553 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:12:06,554 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2571 [2023-09-07 18:12:06,555 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:12:06,555 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:12:06,555 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:12:06,556 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:12:06,556 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:12:06,562 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:12:06,564 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:12:06.564451 [2023-09-07 18:12:06,716 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:06,742 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:12:06,771 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:12:06,773 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:12:06,779 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:12:06,780 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694110326.7405307, 'te': 1694110326.7468522, 'initial_status': True} ====================================================================================================  [2023-09-07 18:12:06,781 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:12:06.781507 [2023-09-07 18:12:06,785 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:12:06,786 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:12:08,972 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:12:08,973 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:12:08,973 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:12:08,973 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:12:08,973 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:12:08,974 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:12:08,975 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:12:08,977 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 18:12:09,017 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:12:09,261 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:09,406 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:09,524 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:12:09,525 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2576 [2023-09-07 18:12:09,525 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:12:09,526 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:12:09,526 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:12:09,526 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:12:09,527 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:12:09,533 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:12:09,536 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:12:09.536061 [2023-09-07 18:12:09,680 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:09,706 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:12:09,736 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:12:09,738 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:12:09,746 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:12:09,746 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694110329.7046654, 'te': 1694110329.7105722, 'initial_status': True} ====================================================================================================  [2023-09-07 18:12:09,748 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:12:09.748108 [2023-09-07 18:12:09,752 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:12:09,753 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:12:11,268 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:12:11,269 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:12:11,270 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 18:12:15,425 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.transactions.tools:190 - Tx alright completed after 1/3 retries [2023-09-07 18:12:15,426 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:320 - sChain: free-basic-rosy-SKALE. 14 node sent an alright note [2023-09-07 18:12:15,451 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110332, schain_finish_ts: 1692377364 [2023-09-07 18:12:17,881 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110332, schain_finish_ts: 1692637668 [2023-09-07 18:12:17,971 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110332, schain_finish_ts: 0 [2023-09-07 18:12:18,016 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:12:18,016 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:12:18,018 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:12:18.018503 [2023-09-07 18:12:18,025 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:12:18,026 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:12:18,031 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:12:18,035 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694110338.024552, 'te': 1694110338.0352254, 'initial_status': True} ====================================================================================================  [2023-09-07 18:12:18,037 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:12:18.037827 [2023-09-07 18:12:18,041 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:12:18,042 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:12:18,070 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:12:18,070 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:12:18,070 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:12:18,070 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:12:18,071 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:12:18,071 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:12:18,072 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:12:18,075 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 18:12:18,095 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:12:18,265 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:18,532 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:18,657 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:12:18,659 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2559 [2023-09-07 18:12:18,660 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:12:18,660 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:12:18,660 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:12:18,661 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:12:18,661 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:12:18,670 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:12:18,673 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:12:18.673249 [2023-09-07 18:12:18,824 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:18,849 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:12:18,875 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:12:18,878 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:12:18,885 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:12:18,885 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694110338.8477218, 'te': 1694110338.8539677, 'initial_status': True} ====================================================================================================  [2023-09-07 18:12:18,887 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:12:18.887195 [2023-09-07 18:12:18,891 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:12:18,892 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:12:23,841 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:12:23,841 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:12:23,841 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 18:12:24,319 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:12:24,320 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:12:24,320 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:12:24,320 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:12:24,320 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:12:24,320 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:12:24,332 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:12:24,336 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 18:12:24,372 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:12:24,586 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:24,794 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:24,905 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:12:24,907 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2574 [2023-09-07 18:12:24,907 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:12:24,908 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:12:24,908 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:12:24,908 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:12:24,909 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:12:24,914 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:12:24,916 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:12:24.916851 [2023-09-07 18:12:25,052 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:25,079 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:12:25,111 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:12:25,112 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:12:25,119 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:12:25,120 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694110345.077171, 'te': 1694110345.0841115, 'initial_status': True} ====================================================================================================  [2023-09-07 18:12:25,121 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:12:25.121478 [2023-09-07 18:12:25,125 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:12:25,126 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:12:25,827 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:12:25,828 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:12:25,833 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:12:25.833119 [2023-09-07 18:12:25,841 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:12:25,841 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:12:25,846 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:12:25,851 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694110345.8395584, 'te': 1694110345.8513231, 'initial_status': True} ====================================================================================================  [2023-09-07 18:12:25,854 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:12:25.854265 [2023-09-07 18:12:25,859 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:12:25,860 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:12:26,014 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110344, schain_finish_ts: 1693742304 [2023-09-07 18:12:31,409 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 18:12:31,411 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 18:12:31,670 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:12:31,670 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:12:31,670 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:12:31,671 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:12:31,671 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:12:31,671 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 18:12:31,695 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:12:31,913 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:32,053 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:32,160 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:12:32,162 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2580 [2023-09-07 18:12:32,162 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:12:32,163 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:12:32,163 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:12:32,163 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:12:32,164 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:12:32,169 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:12:32,171 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:12:32.171828 [2023-09-07 18:12:32,395 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:32,420 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:12:32,449 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:12:32,450 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:12:32,457 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:12:32,458 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694110352.418859, 'te': 1694110352.425075, 'initial_status': True} ====================================================================================================  [2023-09-07 18:12:32,459 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:12:32.459627 [2023-09-07 18:12:32,463 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:12:32,464 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:12:34,571 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:12:34,571 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:12:34,574 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:12:34.574434 [2023-09-07 18:12:34,583 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:12:34,583 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:12:34,590 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:12:34,595 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694110354.5817149, 'te': 1694110354.5951068, 'initial_status': True} ====================================================================================================  [2023-09-07 18:12:34,597 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:12:34.597043 [2023-09-07 18:12:34,601 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:12:34,601 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:12:35,072 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110344, schain_finish_ts: 1693742004 [2023-09-07 18:12:38,859 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:12:38,859 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:12:38,859 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:12:38,859 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:12:38,859 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:12:38,860 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 18:12:38,896 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:12:39,127 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:39,267 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:39,369 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:12:39,370 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2572 [2023-09-07 18:12:39,371 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:12:39,371 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:12:39,372 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:12:39,372 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:12:39,373 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:12:39,384 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:12:39,386 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:12:39.386841 [2023-09-07 18:12:39,805 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:39,834 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:12:39,866 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:12:39,868 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:12:39,876 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:12:39,877 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694110359.8323994, 'te': 1694110359.840341, 'initial_status': True} ====================================================================================================  [2023-09-07 18:12:39,879 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:12:39.879421 [2023-09-07 18:12:39,884 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:12:39,884 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:12:47,737 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110356, schain_finish_ts: 1692637668 [2023-09-07 18:12:50,977 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110368, schain_finish_ts: 1694108304 [2023-09-07 18:12:51,078 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110368, schain_finish_ts: 1693412520 [2023-09-07 18:12:51,325 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:12:51,325 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:12:51,325 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:12:51,326 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:12:51,326 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:12:51,326 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:12:51,328 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:12:51,330 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 18:12:51,355 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:12:51,566 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:51,890 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:51,986 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:12:51,987 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2575 [2023-09-07 18:12:51,987 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:12:51,988 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:12:51,988 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:12:51,988 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:12:51,988 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:12:51,994 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:12:51,996 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:12:51.996805 [2023-09-07 18:12:52,107 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:12:52,108 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:12:52,108 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 18:12:52,140 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 18:12:52,168 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:52,207 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:12:52,223 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:12:52,223 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:12:52,224 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:12:52,224 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:12:52,224 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:12:52,224 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:12:52,226 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:12:52,227 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 18:12:52,248 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:12:52,252 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:12:52,263 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:12:52,263 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694110372.2043288, 'te': 1694110372.2134843, 'initial_status': True} ====================================================================================================  [2023-09-07 18:12:52,265 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:12:52,265 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:12:52.265357 [2023-09-07 18:12:52,270 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:12:52,271 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:12:52,318 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:12:52,318 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:12:52,335 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:12:52,341 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 18:12:52,516 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 18:12:52,516 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 18:12:52,518 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 18:12:52,524 ERROR][11][wan-tasty-kitalpha][T_1] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:12:52,525 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:12:52,620 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:52,764 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:52,872 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:12:52,873 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2577 [2023-09-07 18:12:52,876 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:12:52,876 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:12:52,877 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:12:52,879 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:12:52,881 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:12:52,887 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:12:52,889 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:12:52.889296 [2023-09-07 18:12:53,080 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:53,113 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:12:53,141 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:12:53,144 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:12:53,153 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:12:53,153 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694110373.1109915, 'te': 1694110373.1186085, 'initial_status': True} ====================================================================================================  [2023-09-07 18:12:53,155 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:12:53.155356 [2023-09-07 18:12:53,159 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:12:53,159 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:12:53,185 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:12:53,186 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:12:53,187 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:12:53.187832 [2023-09-07 18:12:53,193 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:12:53,194 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:12:53,200 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:12:53,204 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694110373.1924999, 'te': 1694110373.2046096, 'initial_status': True} ====================================================================================================  [2023-09-07 18:12:53,206 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:12:53.206681 [2023-09-07 18:12:53,212 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:12:53,212 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:12:54,007 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:12:54,007 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:12:54,009 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:12:54.009383 [2023-09-07 18:12:54,016 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:12:54,016 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:12:54,022 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:12:54,028 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694110374.0149305, 'te': 1694110374.027447, 'initial_status': True} ====================================================================================================  [2023-09-07 18:12:54,030 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:12:54.030584 [2023-09-07 18:12:54,030 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110368, schain_finish_ts: 1692377364 [2023-09-07 18:12:54,036 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:12:54,036 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:12:55,163 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:12:55,163 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:12:55,164 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:12:55,164 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:12:55,164 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:12:55,164 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:12:55,166 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:12:55,168 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 18:12:55,189 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:12:55,374 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:55,524 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:55,615 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:12:55,616 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2560 [2023-09-07 18:12:55,616 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:12:55,617 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:12:55,617 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:12:55,617 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:12:55,617 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:12:55,627 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:12:55,629 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:12:55.629413 [2023-09-07 18:12:55,785 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:55,813 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:12:55,860 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:12:55,862 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:12:55,871 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:12:55,871 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694110375.8112035, 'te': 1694110375.8223548, 'initial_status': True} ====================================================================================================  [2023-09-07 18:12:55,873 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:12:55.873263 [2023-09-07 18:12:55,887 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:12:55,888 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:12:56,942 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:12:56,942 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:12:56,944 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:12:56.944763 [2023-09-07 18:12:56,953 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:12:56,953 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:12:56,959 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:12:56,963 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694110376.9514823, 'te': 1694110376.963414, 'initial_status': True} ====================================================================================================  [2023-09-07 18:12:56,965 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:12:56.965136 [2023-09-07 18:12:56,969 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:12:56,970 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:12:57,590 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110368, schain_finish_ts: 1693742304 [2023-09-07 18:12:58,718 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:12:58,718 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:12:58,718 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:12:58,718 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 18:12:58,719 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 18:12:58,719 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:12:58,720 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:12:58,722 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 18:12:58,744 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:12:58,776 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110368, schain_finish_ts: 0 [2023-09-07 18:12:58,958 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:59,144 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:59,275 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:12:59,276 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2581 [2023-09-07 18:12:59,276 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:12:59,277 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:12:59,277 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:12:59,277 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:12:59,278 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:12:59,289 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:12:59,294 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:12:59.294008 [2023-09-07 18:12:59,484 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:12:59,518 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:12:59,549 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:12:59,550 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:12:59,558 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:12:59,558 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694110379.5160003, 'te': 1694110379.5233395, 'initial_status': True} ====================================================================================================  [2023-09-07 18:12:59,560 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:12:59.560216 [2023-09-07 18:12:59,564 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:12:59,564 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:12:59,910 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:12:59,911 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:12:59,911 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 18:13:00,485 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 18:13:00,485 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:13:00,487 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:13:00.487371 [2023-09-07 18:13:00,491 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:13:03,589 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 18:13:03,764 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:13:03,764 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 18:13:03,764 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 18:13:03,765 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 18:13:03,955 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 18:13:03,955 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 18:13:03,962 INFO][8][curly-red-alterf][T_0] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 18:13:03,967 ERROR][8][curly-red-alterf][T_0] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:13:03,968 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:13:04,610 INFO][1][MainProcess][MainThread] - core.schains.process_manager:57 - Process manager started [2023-09-07 18:13:04,829 INFO][1][MainProcess][MainThread] - core.schains.notifications:35 - Node account has 407295140193863165 WEI [2023-09-07 18:13:04,830 INFO][1][MainProcess][MainThread] - tools.notifications.messages:188 - Saving new balance state 3021 1 [2023-09-07 18:13:04,831 INFO][1][MainProcess][MainThread] - core.schains.process_manager:95 - Fetching schains to monitor... [2023-09-07 18:13:10,418 INFO][1][MainProcess][MainThread] - core.schains.process_manager:109 - Get leaving_history for node ... [2023-09-07 18:13:11,794 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110380, schain_finish_ts: 1693742004 [2023-09-07 18:13:12,373 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110380, schain_finish_ts: 1692377616 [2023-09-07 18:13:12,942 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:13:12,942 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:13:12,943 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:13:12,943 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 18:13:12,943 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 18:13:12,943 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:13:12,946 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:13:12,947 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 18:13:12,971 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:13:13,192 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:13:13,376 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:13:13,491 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:13:13,492 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2573 [2023-09-07 18:13:13,493 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:13:13,493 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:13:13,493 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:13:13,493 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:13:13,494 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:13:13,502 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:13:13,505 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:13:13.505001 [2023-09-07 18:13:13,692 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:13:13,718 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:13:13,756 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:13:13,758 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:13:13,766 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:13:13,767 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694110393.7161968, 'te': 1694110393.7234185, 'initial_status': True} ====================================================================================================  [2023-09-07 18:13:13,771 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:13:13.771936 [2023-09-07 18:13:13,776 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:13:13,777 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:13:14,326 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110392, schain_finish_ts: 0 [2023-09-07 18:13:14,694 INFO][1][MainProcess][MainThread] - core.schains.process_manager:117 - Got leaving sChains for the node: [] [2023-09-07 18:13:14,695 INFO][1][MainProcess][MainThread] - core.schains.process_manager:101 -  ==================================================================================================== Monitoring sChains Node ID: 16 sChains on node: [{'name': 'portly-passionate-sirius', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666281384, 'startBlock': 7803380, 'deposit': 0, 'index': 1, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x0000017723ca2', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'curly-red-alterf', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1676058936, 'startBlock': 8469646, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000001fe6cdfa', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'howling-hoedus', 'mainnetOwner': '0x1026467C91fdA40aCc7e6935d1E52f67274FD669', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666181400, 'startBlock': 7796660, 'deposit': 0, 'index': 0, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0xe5a600f4c2608', 'multitransactionMode': False, 'thresholdEncryption': False, 'active': True}, {'name': 'green-giddy-denebola', 'mainnetOwner': '0x8e835d5d9E28ffD259E8Fa8988Cacfc333A582B4', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674565164, 'startBlock': 8368049, 'deposit': 0, 'index': 4, 'generation': 1, 'originator': '0xd2001DAb6898127Be2F167B548691C87251D13C3', 'chainId': '0x00000585eb4b1', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'wan-tasty-kitalpha', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674568392, 'startBlock': 8368271, 'deposit': 0, 'index': 5, 'generation': 1, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x00000152e7be9', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'free-basic-rosy-SKALE', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1693996920, 'startBlock': 9646443, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000002bb90439', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'submissive-well-off-rotanev', 'mainnetOwner': '0x487360568fE5Ee4AE5c75782D5F63e9935B0E512', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1692361104, 'startBlock': 9539053, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000003791fa10', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}] Number of sChains on node: 7 Empty sChain structs: 0 ====================================================================================================  [2023-09-07 18:13:14,710 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 18:13:14,710 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:13:14,712 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:13:14.712319 [2023-09-07 18:13:14,716 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:13:15,063 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:13:15,063 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain portly-passionate-sirius - Process is running: PID = 7 [2023-09-07 18:13:15,433 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:13:15,434 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain curly-red-alterf - Process is running: PID = 8 [2023-09-07 18:13:15,803 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:13:15,803 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain howling-hoedus - Process is running: PID = 9 [2023-09-07 18:13:16,035 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110392, schain_finish_ts: 1694108304 [2023-09-07 18:13:16,180 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:13:16,180 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain green-giddy-denebola - Process is running: PID = 10 [2023-09-07 18:13:16,551 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:13:16,551 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain wan-tasty-kitalpha - Process is running: PID = 11 [2023-09-07 18:13:16,923 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:13:16,923 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain free-basic-rosy-SKALE - Process is running: PID = 30622 [2023-09-07 18:13:17,171 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:13:17,172 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:13:17,172 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 18:13:17,303 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:13:17,303 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain submissive-well-off-rotanev - Process is running: PID = 24912 [2023-09-07 18:13:17,303 INFO][1][MainProcess][MainThread] - core.schains.process_manager:88 - Process manager procedure finished [2023-09-07 18:13:17,303 INFO][1][MainProcess][MainThread] - __main__:65 - Sleeping for 90s after run_process_manager [2023-09-07 18:13:19,398 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 18:13:19,399 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 18:13:19,649 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110392, schain_finish_ts: 1693742304 [2023-09-07 18:13:20,779 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:13:20,779 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:13:20,779 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:13:20,779 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:13:20,780 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:13:20,780 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 18:13:20,808 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:13:21,003 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:13:21,169 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:13:21,284 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:13:21,285 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2582 [2023-09-07 18:13:21,286 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:13:21,286 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:13:21,286 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:13:21,287 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:13:21,287 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:13:21,293 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:13:21,296 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:13:21.296144 [2023-09-07 18:13:21,599 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:13:21,624 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:13:21,660 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:13:21,662 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:13:21,669 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:13:21,669 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694110401.6227071, 'te': 1694110401.6312973, 'initial_status': True} ====================================================================================================  [2023-09-07 18:13:21,671 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:13:21.671478 [2023-09-07 18:13:21,676 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:13:21,677 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:13:25,282 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110392, schain_finish_ts: 1692637668 [2023-09-07 18:13:26,410 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:13:26,410 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:13:26,410 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:13:26,411 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:13:26,411 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:13:26,411 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:13:26,413 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:13:26,416 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 18:13:26,446 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:13:26,683 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:13:26,863 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:13:26,980 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:13:26,981 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2576 [2023-09-07 18:13:26,982 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:13:26,982 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:13:26,983 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:13:26,983 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:13:26,983 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:13:26,990 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:13:26,992 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:13:26.992498 [2023-09-07 18:13:27,189 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110404, schain_finish_ts: 1693412520 [2023-09-07 18:13:27,209 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:13:27,258 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:13:27,305 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:13:27,308 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:13:27,318 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:13:27,318 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694110407.2563932, 'te': 1694110407.265845, 'initial_status': True} ====================================================================================================  [2023-09-07 18:13:27,321 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:13:27.321302 [2023-09-07 18:13:27,328 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:13:27,328 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:13:28,196 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:13:28,196 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:13:28,198 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:13:28.198667 [2023-09-07 18:13:28,208 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:13:28,208 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:13:28,214 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:13:28,219 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694110408.2068508, 'te': 1694110408.2190368, 'initial_status': True} ====================================================================================================  [2023-09-07 18:13:28,220 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:13:28.220979 [2023-09-07 18:13:28,225 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:13:28,226 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:13:28,339 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:13:28,339 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:13:28,339 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:13:28,339 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:13:28,339 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:13:28,339 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:13:28,341 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:13:28,342 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 18:13:28,375 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:13:28,647 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:13:28,843 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:13:28,978 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:13:28,979 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2578 [2023-09-07 18:13:28,980 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:13:28,980 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:13:28,980 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:13:28,980 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:13:28,981 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:13:28,986 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:13:28,988 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:13:28.988680 [2023-09-07 18:13:29,148 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:13:29,181 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:13:29,217 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:13:29,220 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:13:29,231 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:13:29,232 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694110409.1790984, 'te': 1694110409.1866004, 'initial_status': True} ====================================================================================================  [2023-09-07 18:13:29,234 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:13:29.234755 [2023-09-07 18:13:29,241 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:13:29,241 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:13:30,123 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:13:30,124 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:13:30,125 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:13:30.125850 [2023-09-07 18:13:30,133 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:13:30,133 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:13:30,139 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:13:30,143 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694110410.1323516, 'te': 1694110410.1435118, 'initial_status': True} ====================================================================================================  [2023-09-07 18:13:30,147 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:13:30.147174 [2023-09-07 18:13:30,152 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:13:30,152 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:13:33,116 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110404, schain_finish_ts: 1692377364 [2023-09-07 18:13:33,402 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 18:13:33,402 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 18:13:34,254 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:13:34,254 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:13:34,254 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:13:34,254 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:13:34,254 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:13:34,255 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:13:34,258 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:13:34,259 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 18:13:34,282 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:13:34,642 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:13:34,834 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:13:34,956 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:13:34,957 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2561 [2023-09-07 18:13:34,958 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:13:34,958 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:13:34,958 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:13:34,959 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:13:34,959 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:13:34,966 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:13:34,969 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:13:34.969325 [2023-09-07 18:13:35,175 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:13:35,205 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:13:35,235 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:13:35,237 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:13:35,244 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:13:35,244 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694110415.2014022, 'te': 1694110415.2122982, 'initial_status': True} ====================================================================================================  [2023-09-07 18:13:35,246 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:13:35.246255 [2023-09-07 18:13:35,254 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:13:35,255 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:13:36,123 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:13:36,124 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:13:36,126 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:13:36.126213 [2023-09-07 18:13:36,134 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:13:36,134 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:13:36,140 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:13:36,145 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694110416.1327791, 'te': 1694110416.1454957, 'initial_status': True} ====================================================================================================  [2023-09-07 18:13:36,147 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:13:36.147335 [2023-09-07 18:13:36,153 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:13:36,153 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:13:38,848 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110416, schain_finish_ts: 0 [2023-09-07 18:13:38,868 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 18:13:39,023 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:13:39,023 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:13:39,023 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:13:39,024 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 18:13:39,201 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 18:13:39,201 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 18:13:39,204 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 18:13:39,209 ERROR][11][wan-tasty-kitalpha][T_0] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:13:39,210 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:13:39,971 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:13:39,971 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:13:39,972 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 18:13:43,712 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110416, schain_finish_ts: 1693742304 [2023-09-07 18:13:44,839 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:13:44,839 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:13:44,840 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:13:44,840 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 18:13:44,840 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 18:13:44,840 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:13:44,842 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:13:44,845 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 18:13:44,873 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:13:45,089 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:13:45,244 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:13:45,368 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:13:45,370 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2583 [2023-09-07 18:13:45,371 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:13:45,371 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:13:45,371 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:13:45,372 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:13:45,372 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:13:45,378 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:13:45,380 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:13:45.380912 [2023-09-07 18:13:45,616 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:13:45,646 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:13:45,686 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:13:45,688 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:13:45,696 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:13:45,696 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694110425.644401, 'te': 1694110425.6517842, 'initial_status': True} ====================================================================================================  [2023-09-07 18:13:45,698 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:13:45.698820 [2023-09-07 18:13:45,704 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:13:45,704 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:13:46,619 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 18:13:46,619 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:13:46,621 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:13:46.621659 [2023-09-07 18:13:46,627 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:13:49,333 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110428, schain_finish_ts: 1692637668 [2023-09-07 18:13:50,448 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:13:50,448 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:13:50,449 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:13:50,449 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:13:50,449 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:13:50,449 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:13:50,451 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:13:50,452 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 18:13:50,507 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:13:50,958 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:13:51,116 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110428, schain_finish_ts: 1694108304 [2023-09-07 18:13:51,150 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:13:51,250 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:13:51,251 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2577 [2023-09-07 18:13:51,252 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:13:51,252 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:13:51,253 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:13:51,253 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:13:51,253 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:13:51,259 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:13:51,262 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:13:51.262333 [2023-09-07 18:13:51,447 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:13:51,473 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:13:51,511 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:13:51,513 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:13:51,520 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:13:51,521 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694110431.4710743, 'te': 1694110431.479029, 'initial_status': True} ====================================================================================================  [2023-09-07 18:13:51,523 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:13:51.523779 [2023-09-07 18:13:51,529 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:13:51,530 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:13:52,229 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:13:52,229 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:13:52,230 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 18:13:52,305 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:13:52,305 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:13:52,307 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:13:52.307934 [2023-09-07 18:13:52,315 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:13:52,315 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:13:52,325 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:13:52,333 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694110432.3134747, 'te': 1694110432.333429, 'initial_status': True} ====================================================================================================  [2023-09-07 18:13:52,335 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:13:52.335517 [2023-09-07 18:13:52,340 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:13:52,341 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:13:52,594 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 18:13:52,775 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:13:52,777 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 18:13:52,780 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 18:13:52,782 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 18:13:52,971 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 18:13:52,972 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 18:13:52,974 INFO][8][curly-red-alterf][T_1] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 18:13:52,979 ERROR][8][curly-red-alterf][T_1] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:13:52,980 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:13:53,886 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110428, schain_finish_ts: 1693742004 [2023-09-07 18:13:55,016 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:13:55,017 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:13:55,017 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:13:55,017 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:13:55,017 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:13:55,018 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 18:13:55,045 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:13:55,260 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:13:55,489 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:13:55,607 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:13:55,608 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2574 [2023-09-07 18:13:55,609 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:13:55,610 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:13:55,610 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:13:55,611 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:13:55,612 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:13:55,620 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:13:55,623 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:13:55.623184 [2023-09-07 18:13:55,969 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:13:55,999 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:13:56,036 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:13:56,038 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:13:56,046 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:13:56,046 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694110435.9969635, 'te': 1694110436.0050523, 'initial_status': True} ====================================================================================================  [2023-09-07 18:13:56,048 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:13:56.048818 [2023-09-07 18:13:56,054 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:13:56,055 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:13:58,037 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:40 - Collecting rotation history for submissive-well-off-rotanev... [2023-09-07 18:13:59,156 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:50 - Rotation data for submissive-well-off-rotanev: Rotation(leaving_node_id=65, new_node_id=65, freeze_until=1694108100, rotation_counter=8) [2023-09-07 18:14:01,311 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110428, schain_finish_ts: 1693412520 [2023-09-07 18:14:01,889 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110440, schain_finish_ts: 0 [2023-09-07 18:14:02,415 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:14:02,415 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:14:02,415 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:14:02,416 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:14:02,416 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:14:02,416 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:14:02,420 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:14:02,423 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 18:14:02,443 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:14:02,768 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:14:02,932 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:14:02,992 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:14:02,992 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:14:02,992 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 18:14:03,031 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:14:03,033 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2579 [2023-09-07 18:14:03,033 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:14:03,034 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:14:03,034 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:14:03,034 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:14:03,034 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:14:03,040 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:14:03,042 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:14:03.042732 [2023-09-07 18:14:03,232 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:14:03,257 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:14:03,283 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:14:03,285 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:14:03,293 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:14:03,294 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694110443.2558024, 'te': 1694110443.2625725, 'initial_status': True} ====================================================================================================  [2023-09-07 18:14:03,295 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:14:03.295629 [2023-09-07 18:14:03,301 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:14:03,301 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:14:04,227 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:14:04,228 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:14:04,230 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:14:04.230553 [2023-09-07 18:14:04,240 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:14:04,240 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:14:04,248 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:14:04,253 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694110444.2371092, 'te': 1694110444.2535794, 'initial_status': True} ====================================================================================================  [2023-09-07 18:14:04,256 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:14:04.256359 [2023-09-07 18:14:04,261 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:14:04,261 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:14:05,542 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 18:14:05,542 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 18:14:10,781 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110440, schain_finish_ts: 1693742304 [2023-09-07 18:14:11,912 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:14:11,913 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:14:11,913 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:14:11,913 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:14:11,913 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:14:11,914 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 18:14:11,933 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:14:12,134 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:14:12,279 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:14:12,385 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:14:12,386 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2584 [2023-09-07 18:14:12,387 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:14:12,387 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:14:12,387 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:14:12,388 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:14:12,388 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:14:12,394 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:14:12,401 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:14:12.401502 [2023-09-07 18:14:12,757 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:14:12,791 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:14:12,821 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:14:12,823 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:14:12,833 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:14:12,834 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694110452.788186, 'te': 1694110452.796406, 'initial_status': True} ====================================================================================================  [2023-09-07 18:14:12,837 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:14:12.837299 [2023-09-07 18:14:12,841 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:14:12,843 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:14:14,213 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110452, schain_finish_ts: 1692377364 [2023-09-07 18:14:14,409 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 63 -> 65 [2023-09-07 18:14:15,339 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:14:15,339 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:14:15,340 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:14:15,340 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:14:15,340 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:14:15,340 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:14:15,342 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:14:15,342 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 18:14:15,369 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:14:15,585 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:14:15,746 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:14:15,869 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:14:15,871 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2562 [2023-09-07 18:14:15,872 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:14:15,872 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:14:15,873 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:14:15,873 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:14:15,873 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:14:15,884 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:14:15,886 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:14:15.886471 [2023-09-07 18:14:16,126 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:14:16,167 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:14:16,198 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:14:16,200 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:14:16,206 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:14:16,207 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694110456.1658447, 'te': 1694110456.17335, 'initial_status': True} ====================================================================================================  [2023-09-07 18:14:16,208 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:14:16.208922 [2023-09-07 18:14:16,212 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:14:16,213 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:14:17,128 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:14:17,129 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:14:17,132 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:14:17.132320 [2023-09-07 18:14:17,141 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:14:17,141 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:14:17,152 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:14:17,159 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694110457.138006, 'te': 1694110457.1591265, 'initial_status': True} ====================================================================================================  [2023-09-07 18:14:17,161 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:14:17.161719 [2023-09-07 18:14:17,166 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:14:17,166 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:14:18,424 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.main:118 - sChain: free-basic-rosy-SKALE: Everyone sent alright [2023-09-07 18:14:18,424 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:18860195493846376099765944407454631582519828060563062387235736098366005:NODE_ID:14:DKG_ID:3'}} [2023-09-07 18:14:18,599 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:14:18,599 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:18860195493846376099765944407454631582519828060563062387235736098366005:NODE_ID:14:DKG_ID:3, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:18860195493846376099765944407454631582519828060563062387235736098366005:NODE_ID:14:DKG_ID:3 [2023-09-07 18:14:18,600 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:280 - sChain: free-basic-rosy-SKALE. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:18860195493846376099765944407454631582519828060563062387235736098366005:NODE_ID:14:DKG_ID:3 [2023-09-07 18:14:18,600 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:18860195493846376099765944407454631582519828060563062387235736098366005:NODE_ID:14:DKG_ID:3', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:18860195493846376099765944407454631582519828060563062387235736098366005:NODE_ID:14:DKG_ID:3', 'ethKeyName': '[SGX_KEY]', 'secretShare': 'd0a5e68a2967ce0e8a4ae7502641dcc508f0edab2535e3e8f95d580e3660524f4036b654f44e5eb0891c418ac60707b46ea7690f10dbb73e58adfdfb4863718cdb2acbecfffe31a9be792b90ad04eaa7b41176d2b2720a9872321f06b8ba6ab240e3129b4afe2bcbc25a40325bba9833e51e628bf244bb0a84dd92a7d408c160f7be3b07d88ca63d543fdde39485ae1c14ea30dabd198f2fd378165f603246ab2b4c9f3ca16bb7086f34da55f9ad94c09c9e19e3039e95e0ffaf20aabcaeecb9f40a45c366c25d183b4b0736591bdf557cddf67e5a50831226ebe1ca489e673222be2d28ea3ee4a31286742063de66e12a0e54448630cc9b97d8ee1fd10e6b83384d08678206811430c379b1196ee55d86817ac4c2c4e6b9007eebf68fcaf8f29ddfa4b94ba159b12dd35e7f545a4a443fd418766ab26ef29dec1655a1de9fe5e9d3f19db537e31f12493717bf0649c5a55ada19d3420dc2ef2b6ccf83f559dbec8471301e057ed3b48d15934724db3a12641ad786739dac5393b5da95e9d94252a09a33fd6943bef89569d547a4bea2bfcab6c3f81cd8451fc2d16a30615bd5e608e3325e876fa92b101ded296cfb26a5b3d7588a61a76b23c1968e1553ab4bb0a208b46f02411358eaf64e961a476b3e2049272effbce765dfa0a77b65fbe5aae43ea2dff9180b132307a3f7897b514b6e1ee2c941f3d8267e3023095dacc524d66c0a10b781db24615cddb6a7189a39785627f70fca3fc9947b9a1f8f258c1cb7946c41f99f65364f78dfac67a53cdfc9cf427e6c296e7b6c6d90881ce7b6b7b89f5e8cedf7ae09c58564073a779471b0089eda91a803a44779b948e35505526be4223b12d4f81f5779a49889d8eff16f6fa3dbc835946fc3578f6e31ab69c011f0e33d4a9dbed4c492d10d37919099bdb0b0e5a4849b2e0e3ed4ff9df07472afd8ec526a3d567c6787938787d0c5e82dd8195cace0ff363750fd87e766ff969a9ef119b2ffdadfd711c5cc88bb546ab8c559a994e7e92292bf786e1df3cc7ae6e5679746db328ff5e6bb9bf5853a1dabbe325b0b79a6167fd31fa3764f6d8392cf9d1b8a4b68597daadef5fec8672b9f41293935afeaebc609964e5f7bb83a6bb0cb18d82d9340e4ce4fd02fd3730ad5d3431959c4aff9dd31d0dbac2fbde9cb6fa5a591d22749675d9b4c1005694c9af4de3cb9e50754f4b9afa312af8c8138363dd7e0ef27feb8fcbeadaa3f7468d831c4b4e109e16f74281e2e00da616810e318b0ca18312a69be474f8545f7293b8b791530004c5f51775f06ad3613f3eb3b0b45c810418d5e93e3b5c3ca0fa7816f9650e5501fb9a6c8a5c0d0453bb58e0a8ed679ca4516031ff90447e3e639a6e49d4a6ca342cb2afc735dbef1ab626dc682b4b58668de23925f911de6abee7bd7c3bff44736c470876f01ed54f1b5f246695bc75b72940aa0c9e9f70625fab91352f0a2aafef74bffd401020532609b42c034fafa9ac51e67056119ae8d23bbddb82d69ef1f24f469cc73a8c19dd95caca9e6f138c6aaf10bf13d66abd3ad3f73c0c909b47fc011d343c319f2820cd8ad7b366e8740f763262a87391c606610840ed52c40eb8091f7a0443fce693893343bdf010c844c57d88684fe96b2708d3e9c16c7b56de670807816bc6fea9c8993d16dc0fab4369251ff6a7e2cf64dd57b78d861b7942b77fd1f07df82b7fa5da6f352e18df4be5fbb96e7f5c497d018605c32a14579ac8573476ee635cb4c9eb30ae6b6e51c421ca69a5bd10de76e2a36bdc927f0a2a89d44b0137aff09468e932ab75180073401daac535ab5448e7511341e6148a6c6247b5bbd641e7e1bc43013afca8349bf8395e22ee87b6df583c7eda8830b09d1cfd1b35069e602ec76e8ef165b475d47c9ad85b9a2b1a55c8f18ab17dae2d32b7951377cb126b7f003eb74b8a82cca3241cbc331d14be8789db2f65e37b5a0e7bf25dbdabee4782c90f7d5b286ca2d4fad9ae26c440394d98cc7773429d27b7940392c2a6d0bd4b10ecebe641f78186dcd7f1ac141c907a8fe67e19075642a4c3355ba590b2089ca0245b7f3eb12b77a349f6df4709aba24c5e614afc9d12bee85671063794050f018abb1a3861b0b52624489e1a7fc3ef891f979d6a3573d614e4920cb9ee7cc', 'n': 16, 't': 11}} [2023-09-07 18:14:18,857 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'status': 0}} [2023-09-07 18:14:18,857 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.client:285 - sChain: free-basic-rosy-SKALE. DKGClient is going to fetch BLS public key with name {self.bls_name} [2023-09-07 18:14:18,858 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:18860195493846376099765944407454631582519828060563062387235736098366005:NODE_ID:14:DKG_ID:3'}} [2023-09-07 18:14:19,023 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'blsPublicKeyShare': ['19259382161055078898663097663288417936824144611507279523322171166920908033289', '14109503097051852876474699521907193802675114999675443259812001025711206550131', '5158117661403121054086554735901161287570541058326318635217612866527395492200', '3715322380648724761584242139609935920634736776088826855208563759886249878624'], 'errorMessage': '', 'status': 0}} [2023-09-07 18:14:19,023 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.dkg.utils:150 - sChain: free-basic-rosy-SKALE. Node`s encrypted bls key is: True [2023-09-07 18:14:19,024 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'calculateAllBLSPublicKeys', 'params': {'t': 11, 'n': 16, 'publicShares': ['06f940322af0e17dfc73a4d839d11d5c89f2f7cebd32a6d3f821fae9ee39b69d122475e87c4caa82d10a44b74c6394f1c087915ef78c63de8b961c475bcabd521b41668ec7b10d510d7548c2cec4d48383c73b5531640df72b07c06a272cb0e921546df2bec89a021bca1a9fae56a0239179a7c19453a800503a669064fb4e7228638743f93615ba3bd8970646910095bf604e23a56c3734a1016d43e937e4d019a01ea6fbb76d7a5ff29c65c0b89642a0f7889e64e61de230479cee5565ab75153de9011900f07d55c11996aca945ff0b71fb646a6d439bc6942b36ba98d31b21353291b7e3f44638f8cefc5026bdefcb4efa5d12d6600881628fc28908098d2ff4f3671fd3a9c5eb735c55235a19b64d811f0ac79f38a2a468e99a07469d652d3b0ef8af98cb11289a2b0bd15e98b151403bda6a614b36f0dd0f4468f16ccb1b322a903ae57d2dcce8d52f5f650d9a8a95ce58effaaef3e5c86110e97e2b6527bba0ca5dc98fb78ddb900e0c04b5497f348cc3540398f296857ac2fb62574f1ce2fc82fed4cb5722f2e2fa5c4a3a45a9c98951ed73b98cf8cd4eb52d6f00a30e56798f9c32d99d29b78559089c68021e90073f95d7ccdf1ca3fa1ff74d524008651f040e65515bb2f43368bda4d7361fd07de9cfec7761f53fe133f5c0b4ba2e1d2b5b06e025e488aa6237b3e9ef06635b766566146fc259b543c0fcd376a10d35a89fc946589a04e3e4e176b8f7401c44b304c2820eec32e7b9182c2c4b301b3884bb287e5fd6b60fe2d1734d307b93a9fc68047226789b1447799ab030ea16c955c0a9427e6bc46a27007ff123b2157818cc975c6cae210752d0e32d128229260d0eb02f9d83560928ea79b6c91081ed8e9e95f2270ec69f50a30084c9e1245c889fc10833d1931dab324b51d1bef7a835521e8d549ee70c92beba9770ab01c3f6a975af0a0d0fc7ada73f2ba4793a99aafd433226ab1e2165e42f3740442b1a5cd7c2e8e91c17b758ea6e8d2feaac2c42c5438c4e5208b6a8a4f77aede304a5d8c57a527343a425115a547e7d506f5a013cdc1046de1d3cac12bf6cd15d18be1c8338e029e1c5a34b071f51fbf023e9af8b098cf790d2ebb55ea9f063cf282d4d2e15594bc4bc8d726b0da761034a4dda55f517e9d6bd8ad1d2b48466a72a6345da244fe023a8f8c3a4e5c53219928b935abbe7045f38222adef4bf2f7a0071355ed76f77c222b774d1bbe24323d71240aa3e135d6e5e56d144c6b705f807ca4fe1c223ae4c8d02384cb70dd5467ee4f18d5dd3cf0155216036a3d4d8d31e4a8ec7aa310444ea8503b166028f836c1524436a4d35f7d1d90802508bbff12d9ee452f7b086be82e2254c28e5fe76026e224713812bf42a692c1b82573e670c344a4ff8f85f6c309b895a075d29429d782160cc046d418c6fb0249e9889390ff16dd04b26f3f49d3c53d7ff5e81a35a7c04fc48dc6e30d69cbd5da25afc0f2500e62ed9032722737ef6f5c61bc715b9e21149d3804f2960c3e250f4c1cb060ab2c46a06519e18e973fd2a62b058c4b460441970a125b22b82dcff99cc25c008d6ca59f5c362903fc39c7aef7563ac127a1705db43035370b8cc3ed5947b9f14bc20b2d106ebce90966f0c338da4bf4cb09a6c60f5005c2cf60f90e82f525b280ff3bffb29b11ecaf7a63aeea9338350e6bd7b84207c3bc532732fdefc9d390300e95a4ee3360569c3a147c31490fc118391dc7ffa9b80344fd9a52b99a3c4279929e7047333205aed2a4ba8b4b564d900f90b58477c2d131a5782a6045b082f7d3fcc7512ed5e4aa4d711f8b88e11989157f280cb021b1b028cae9805279f1c2b06c627f6cbaa6efb0bd9201131f908fcd18e470c2a7ff3aea346d826dd8a24e9b5579e751b922a495ae7cb268773e54bad30e2f9b19c0c32072adc1583bb30132ae8586e423788c2128c3e774ee30bb0342c2e3e1315c4d62fcc96f94c1e', '165fbbf7b02bda9a8df005ac2d84332aa08bb43d47ffd79c6eb3bd705c5ec40c24e241fd220794ed87ec5f48a1c3379f235ece8ecb5cb1bbcd76d05d07da85e7051fea135cedb84e940dde6b6fc1c5476a0057023c0b10c4377948d7348e69c51e79984fc43e3a1983829e3e2c50685605a9c39b45e5c00a3794c5c337ccab3b071bfdd501f730df68da271f4274cfeac374477416f990a90b90f94518ebbb0722843f2d6377fa7fad202f1cd7b22f438a4ae96ee38c0e6f595b3be1668fe08b00d1fbd53c4ecf9abc928a5d055b70ee9b65411f19809f41e7ac4cfee8a11b872805cea45eba2735cc96522adc1c7c24efde503664a98d4083b2bf3f9a9151622e798a5f9e7e51a02e1576aa9f48b82329953f759275e9b2653c2c6d9478a39006548eb841f0fe505170b52db218194854ddecfb11c145846724699c7cbc12bb20f72b1ef7194c061e232afddc0c39cb132cbf5feb8b594de362e9d45cf668450f711a63f72f2bd3242a227e573b727742b665a4de0399460be390a1309d763606d2bca2bbd307e0213a4da1de81907c2503d8d593c6c7ffd3093b51b9d44d5e14cdf0c394059600047d8ebc84b0e30df3b0523f2c03a65f0fecd054a5fb8ce717ad56789c461ccafa58f91a4418f0455c836cf5e7151001fc313696e7aa7a22150077242d0414f5d9cb209895693129c5afd4be321f056c9ca9dd668d29c34903fcece200fb2e36aa314867be9b6024f55ce5da9cc529b875795f03dc75199c2c39d96abbfffc3e0c47ce28d4a73f3ce95a84de6a7cd48bcce1713f53da20001cfa9d2c7546cccad3f79a53bfbbc29798808856aa32a59ca14bfed9587c3c060a225b66a24e3bf84180ad7b63b13f56c3a07c887a6c0e8377714a4f00f5a0101b5f8378c1e0b664d19fb274251e8d1e798be866df7e7b80aac447637d81c73d22e4d8903a8faed7c600ca5eab09eb6ff9666ce2fecb2a752e5c7d6feeb520062e6615ce1ed93fd4930c118c164a072181648d2fc9816b8d5b04c7c0f5ecdb660bc79ec4117190545d88b4800f54e3fe6d84855d5319b717d1de888816becc012b43fedf440562cf65d19958276a4edf7a2253906324d37d5ec0dff45d511e8d13900f1e678bed3b6b74ac76c25351c6174cfd9fb60a970b23c178e71944bf0e05030e7070cf5180f29bf52b65396badc3e971604d8ca61f924d2c8223d8cf4006523f5fe0027aa6d4bac1953425e54a19a31bd0b8f8cd879ed4ca0ec3d372d903cbf2aeb95325e78fd88eb8fd156633c8d5d3d7b4d211fe4c75a3cad4d3180c20e775f5397a54c0aaa7f7b7fcf60553e5c12df8a07cb51ea06c1df50548cb1d006e03acdbb016f578ce4b16951a543030a094b65412deef72ba331d5c84bf9f1c410f01688abb2ebadf78863f22578e80fa53678109a051ff715ade6173eccc171ddf9c394aeb2e07da624ec08919f5707878f7e26b8af85d3c66558d4e6bde0999dd585895ea616539bdd91f9113f1d4dca777a8b35a379838fdeeb5ff8c3b032091233859d78247ba9c84d61c55bd823cbe60fae8bcc0d67a7f19e5949de512d6e340deab52eee7679c7370133f3e18e8b2107ba50fa63af9a06a2d189a3a149650bb126af853e0457beeefaafb9a81e23fdb08d07f808d7710ce934f28731c2a043e9c0fdb2f1cc53b5cc37de5d14c38592a960afd04998eac5b45d37168285774507c75e99867f2a6f751d059f9a6968d35c33ccf127ddf1ffcc168966d22272adbe4d82a550bfa0b50ca5b3b8da1a96f0a9c723d664ce103884401c74e27988fd9aeac932852615d053ba2ee79043683c9a157d6f7b4bd9493f01e86a22f38d89f887c8832c5e73ff4d1180048a23e98ce0258d7d1bdd31b2adc58b36605fa3fb71f4976d9a8ddd96e30cc5a339ebeefb1d8d61c24bc0d8c5902a5533019c5c6f48d0ac028b88c33ad6ebf498b505581b304c74ae7d9fcd11faab6db1d', '1f30027978341f8e1c397447d47208665ca24199f4364c22e8378f1c13769c88099bcd5a18e7729cb269f7c590d41e3600d0fc190402bc8298928bab6c0aee500e332874c426181e43db9714e090e72140d95445accf5b41f110f9eb719118851f0d267f4ba8f2d910d81f1eeb226b48f2e6142004045fc30b6963983651ba4703475aa22ebd6b6ca9f3aba83e288977bd2f73c925f6a0bab2e8d404aea9c4e9271d8fe5a9ebc375d6b974f43bad6ff923283a77605e5dc88e5b554a2f4a1e1f051c8d8813f5053bab76737835d1f20f63b3471efd693265f2702ea30efac39828a3e5b167a3d3b8c9b048c9f93e45434602bd92d0da9f0352598d491e451d0b01da171560275464a0bee6d96211423c955753a2d049845e671b5c6436cf4081185df4ee2772792491e555e056584964cf3758d027318d01a95dc96154e7aca9110b5ac02e74dfee2fd7f924776328ebef8822a877424615b31c0209dffecb4724e1c2f270c91c0d0b3240c67bbf9c009d1bebfce152063dbc2e1013b4d3959f17d271efbe637c1348f8dad5368dda11b1dad07c8348c1cccd415a39ff0ad31d2518d08b5c3c566e109ae0b6de86027020d7e754ee4e9ffebe206e3e28f9e50429669b5229f1cc9042e3d368f4ff8c9b8d4f0d070adae51c24ff9677bb7b4e4d22015079e1ba4d1de04bbc63564623ed42fc62d7a3beac760f69fd54345ebf64303f323a4fd287a2dfd6858c8c685c0103db575c73a32ec69e2d9f31a6c09cba14397179e26e3b743770a2483504a712ef55ac26c8815c46fb6c17c1dea7f24c0c2091db81ca4ac226db927bbb540b0af72bf021f32432a4568834f706f3b3811797da5be9a0d619c4d29dd295193724b6e605993e19ae8d4345e7ff41c83b2a0822da7240e1ebd5a3dbc8415e944dec5bde3236da6e790d60cbc93c9da2d8f62af6a08b1c682bcb0c2e3b3e5324e71e7978aa4de22dd24bd9c590109352683902026f0773f8e693cad2da33354223a6934a99b4cbfc89ffa3a24e2666351cdb1e2c1071ffaca9643a4111da86a7b10434c531b32851493e56423a061d17c45721a1ef183c04b597551d9a460fce77cf7d34464debe2abd91983797abbe9a92215dc35f15a48d143dd74ee322657d033c0fba58b99e908d37c426504645ff95818e5e00c4c1b273aedae625864fee0ad9b7cf4fe66214b9ee6041d70f661b05016c7070f37fec93fbf3c02184c4f5172cbe74ec3aee274b2e9d96572afd1fd550fe7d9257ab21a5e3fc0bd1c2d04908c745944fb6b9e7772e87f4a44287fd5b829662c5ac4b23c749e9fc342fd9eaca95132819337d662863eb495175bc89d9723a384f2d3852c354dfd557589fc53714622e0ea6cd88f4ed6125734b41ce2cb16a585dcba07521810a24a9a2d20fecfed927c6d0a1d0e91c5c324cb7fc7c08d1b03d209b19c539a4e723b34b927ef5da87a03c6868745263dff3a801b3a8fad2bda2111688c0d0181fd4e3e7b386c988ee176fabf42d6a92c1f587957b7a12d18a0b29860c39a6a2ab836e79cf5c54770f9c2e57c32c1fddcb8cf9c06e7389d1917d318c11581b97bbd43eab526b1d277260e31b57ce7f5cce78bf52e4263ff0ec4da50e5b21fd5b2dc994db0af78d826b0f254d78c4b33ce844f9b16adc19908d69ffead75cf3663aedbf48d71ddba753997d021f2580c0b3a7a6c6c25b4f3304fcd47786793b66a60da991fc2665a5712f63f6e7f31662678ec3960d3e9e1114d022afc2153e7945a787d68b9f14ae6cb72e505ffb1b1cfc523db6d8554311062df021db4b63431a2afc6eaf5e08a61879b3d781bc09c479f7811c1c229dd0cf4073adf7323b1328e00b0c64b56d9c96b06a11152e43a90c0e618f957b7e00bf0eae5706a73efe7fda65cde64e342c635af0762216ba8c23e5e68e09fb4791c3617d51fc205a485f5d790c6e08e07f68267485103d27dd23f16f0e673bdbb', '0204f8c10ed0f364d22fe764dac79c5ae597b513de638f22179e6f1b2bcd40c1269a615fefc5dfe836e64b287d7a693fe67622f6e99c11956fd7db0cadaec0420d1511c318231c3e6f2629db68588ae4055e92559b8f59f9e33a30f1bd3782ac0ef6a4b4a0f1209b2bb4b3ad1aa55b32eb9d765bea71eb7079e49ea6dc186529138c43fb4347750ef2c399d02ddf0294014af29253a01351d80866765a8172442e8aac314b1e0e49e14fbdbf6a6acee7a0cbc0ed6b62609f742f6c873651550a1b590522388468d4eff8e9e3e061c05417d0a2669db5bdd7587e2d5d857268d00d56931374464b79cd56994fa04e59eec02910862aa0824ddf291e0c143d2f9423725b0a8236f7b0fd8a87953370aab1fb8e96bb14b475f59b1f8fc90b5711cf11af3406a91033560f6c4200442d9d9a5388520b67dd258ef7dc3c68a93dc3fa13949dbdcbe9ea0c6047959ee9b1cab432a65e82f4b8171ec21b4117d8c0c2b5066c29f787af33c9023047714c613e4545827097609b5cce35abab071193c76206ab079b748b891296faf4ea0d6de582071636284b127471493645f4037ce444039f705b2108414875ccee791499fc1a32bfa78cdd69781f975c54b63d82d2201687d21b33971f0e452038f35d02ae078f963a6d2dd4a37c3fcfd0fb1e4990d4145c344425f7444a6e2087a79dfd5064a7d87cd708493b477b36c88beb5021ab17411163952c82f56cba763885d0d6838773c256c07e3264e1c379c1e93509122853e22b3d0491779c1f5aa380145f46397962a04ba489e0a619559744a8777b23b0dcfdab88c5c4b1c01d8e918fa7792ca70fbd5778f25b6327d58ca8f1f51f2212965e886c3c9b5321fcbe747dad127d7e0f4033c7f1d1c99d01465a2e07d52281e6d9039ee28999a0fc126721740cf7a0cb34b716cd01a12803a67a9a7d7d1996bae5c4cff1c0f0dfa662d6340221a4c2d06fed1d4e4fa0d1f1f915b5ed3528f78de51ebe23f4d0f7ce51aa624d52f2589698eef65133a818dc1ffadc6e100cf1824e3033d60fdb138f9b3d7a1dc20fa55a227ee8f95a1671213f0363a0da0efd3b8e774e28208fe0057e1bfa32fd093bdfcfb3a87c54b9c82335c0d06a2c0d798dc88367f62c66c0d0a6ed717e0333fc52ab9d8b19017909a38f10403f41199e7b37b13974e977d6840108d6a54bec2629a3dc4d2a4f8724c74ba7bb795c1e6054cb69f88b4a3e750fca7d0a13b34184273c6877d8b05077d626fe38e30d162a6306825e03992fb6d73bb341f7bff8527d49b755f24cbdbba390d57f34e310b8256ecea7201f766733156a88051f8bc9688095676eaee1ec38afcaf767e42788012e262aba6ca205da6c3cc2232c32768802738e85ffbe7cd9fb4ff2f41601d8f7116a965b67220077f506a04f5d7d5a828990071a5bccff756f0d4a86a1031016be44883e0766db9a919260412c8fa6fda85590184dd6a3bb829eecb1f00d0646746ad54e2262267bc2b6dcdd1e0f8845784685995dcf2f6a8434fe6bbe18b3b80f363134496a58414323916d8b25ddcdb9bea2f90232e2a0474bc5656925ba76345180e05878e9144f61fc52a3bc1885414442fc51fb2787c6080a508914fe3f7af7a6919eef04ddb6b9c19662c9aa3124f0633080a87de57a5fb4f4500296a6f968267e3c4ddf1b3a63c04997983f0b4bfb6fb4ae02428d0e751b0dea1a3347e65ce6ba65b6302c12e1a5e4a5c8aee344adbab4f1944051f0e84b807006bef92bd85fd51a085127f4b1701e33dc1e08ad253f68399285138dafd1c0790e8185940355c0e3379997dba7b6dc01048cbfe8b1c3536ef6bea9dd3389a8f22258e6bf98ebd3d63f45d8db63ca38dfa6a0267b8bb2796907d879c4446dafbe1da75420ca9189a4f4c87fa53904eb97acdef402c3603293f963fd110e3400301e1bb7b2e33b2e894d567d9a631470e673c52db2c376f1ad44f794c3adec161e', '11d3ed5719c1134a265dde10aad060dffe8d0c15cc37208f877afb27df60d63c1feb78f5dd690f58252dc4c8b36d2a3a759ddaf2ced8e00da00f775c827b94c225a0d37e56bf165cdf21cb9d4ae1c5b70658a0f62a15f891c44c458d17f151671727ca4a132435e46d551e7b4943c5b5ff59bacfc6c696a0b921ee8dbb00922e01a4e75feb7e6e421ddb0f0afcb130b3d464f04abd64e3765785c7511e0174de2a5ab7f5e763e25fabd0f8f25a588b636f51b49d0ba81fd4948ddf57bf6ffb22102d3328b031c2bc53000b85c34b1c119724ed3ef480bcfc27ab88dad6123a44156b983ec34d4715008d8e7465bfcb5c6c112a50a566fb780235c45b3985b1222cf93c17f2d8a261bf9bbba8b197f28c8fbbea4f2ac09548b128db708a426e0222dbed632f3a388fe4856680ab136be722d739a3f759438de814b66636d4862806714b6de9b3e8c4cf425b94e3535846bd887245b0344f2abb43f58841447db7122dfe1891abb1d8a67527ef67b85442870ac750e46d622fc04df90ea3af48a00b2718153cefc172c03c6a33239fd9d7fc5ca2d6217c5f05ce6a98035aeabd600fa5648a0970d264d6fa60663e1d61af31da352f512d4024fe3fadfba0536bc523f976814d5bf431707c870fa414cad0e313d89c65d37c32ae25d033a37ea77b1ba989f9dae2ab4c4ac1a7b511bd72373bd72cd642617a555c303ccea251dfc50d70f26855cc566780161e6ff7d11d9726dd441e1e614c388bbcc61cc7bc15862cdc0f4f82f1feac14ff60cad2fc1dc95d492d3d1df2153622eff3b73dc0fa5b120017e57e8992eddee17ce237ee455227319cfbf27a666671ca9a59966a29742ad883c249ba666e1da99af09c823bf064ec2bd3fc01db32d38992674494568a08fa9ef75032d0836bad934cf57884002292886dbaf77f0d8aef60cb3d7844ee008acd714d4da0a32335559be96c1626da7fa483ae815c4a98955dd1be7a5dad0d2b9bd8c55090cbb957db3d32688ff50c7c714862c2b0f392da25be78ab1f142061a072ae4805e062da8cab266d478fd56d9464a068328a4b2a02970e22858e28dbe11449b68f1c5ed78674c4981f7bc8911d52578db642d8e876694f282b302281699d08b34cc7dbedc3daef74e6ce9d43a436a41699e563cede852f8f313d03d7aa130782d4785af24dd3af44abe63575f50f5a52efa91cff0296a125fc4d1e13915a8e03c265dbd384ec211b9688a48a7c280082963750ff4220335f2227022e0792df337105312aa1b23fa5330541a37857125f32e76e050f29a66cb46c0f0ba631cd9b002b852ccb11d4ff1e65a8d88a029db88d727b32eb13dbdd34051964e121aa8bb9f565250177f59bbdeb46b674d5963109c7adf2c7546a34252b094fedbcf65b716e5183bb9281d95b6edbe2388a3400c62645b7361cca6eddd227e945d0960427b205debf5d77814dcea6bcf9e91addc8b3bd89b019f6aa007a19ca7f8f41f506b4c03ddcba669b800a753f45a1458a0a4f4f9440a5660094c62925c06f42a72f3e16beafcbfee07109d5514ac748947ade43a66c10cd749270105260b8b48d42549ceb8819d9fa206a072f3ad54ab02b257b824009374c43a212a026d6b1f9801cbcb3e60b015b1ed7faa114a92ca1d3be7076d1fe049302010efdef319de9461d8053f43284ac765506968cfe4c0ee1b430eb53131915a33a1e4aafa8b1c57126c108357eb1bc23a511497b7ad74f4b893ae14457f8854a321df37b341dbe82031eadcced3fba7f4f01c2656713d0152efea4febff23b43bf121a8e1a3f728d659f8a4e1ac2d6c2e50be55342c89fe6a13a7c9c8308bd662b2ce4a577afcfef4d3256e7c252e227bcd87bbbbdfd3ee1c772a018194e0fd104295e3af903938a07007d20b3d370c7c1af505c9ac8b6809e938e1a99d2be2b1419acd860f66631f30bcc1afbdf004e56853930384f0a0e0abb1b85d75dd72307', '20f09539483b5a3cfc5c8dfe0f92d8d7311b96849d2138e55362ee8184bfdefe124fc3e445d7cc4f2bffa9c645bd0ad32e56b094fe602cfb42b93be1cf882c3404a6d9ded4578c11519eb9dc188388d35208a80884c8d7a5c1999952c9bf3892133fa2bfbaeb8b926891e176f13ccba1ee679bd7b8231f0a237e3d5741df343a15841c4ce4d081dd97a5fd79acc0c20677b1640e717b3c7ee9344f4ab4aac56500f556e6b22b10b6b490d77bb8fb35d2eded2e15b2c9eda331f6ab7c1405546c220efdb1995e8a0247fcdd6c0f8d01c34f6affed7416899d3161bd9cbfe0ff8f1a7adcc6d36c66cf468bec8491acf6f04e0d4a972d87afa49b010f063507f50d2b384eb4dce41a01825d03a58696da400610f4dc2a256b2b1e141f64296f7fa616536e3e29341545a2125e52c12722f8a73824e41746691f537e33a62dd692180dde65337985d924f0c71d3cd35d176bc3574cfa20ba06f6c60878663cbdb9bb0c8425cec63ecfb0d2f7ba49ddfb82b3c5ac63dffe91f3eaa555467144b80b460c5a6fc4cc737d9d21635fc0ffd09b4a4407e40ed8545df8d9f2c6ba9696b213242f5cc9e19bcae8369174a0a276b0c73693a8f3af66029f385b623580a4e32e155b55de42dcbac7e085dc7b631e723e3259ef39ad101a82e8221c56ec163e6d0a75436df7bde1823bfa9e31ae770df427b3c27c498ab8ec5c0ea0296351046a14e9525ee73d94eee29dd40a82e7e0ee072d8fce9eecd2ff4e7bf602cc0cd18c0ad790340fb9476e66460ed7a8b9e033152a2326b518044cb7a2b05be601b86d0ff770d6155a9659d1729503854907f5751ce44ca841d2aabbc80630100de2d92b8cce7bddb0bf81fda9065aa6b0df299a802307dd52bfccb817d15d4629b41a25994e25681e7be30c4bea2502d8c7f8be22308365f5b6bd96a4aaa1f68a5ce60c170e435caab6e8f729ee57f046c10602fb261d59422d2fc686f418a86f3ebf0d9d92a2c494b8d8f12a2511f2ac7f4fff6df467f154e29b1f5be7d94824b9de14805b99505b782588b334b027b63ea4582c8a4e866e5419f3044e3a41b09d4e22aed8fadc8acf6d3e624edaa64541b8507909e46b41f0ad187fff72ed0894b40539086c2547277360438e412b62693911bb13e7a7cd4e380b010cdff1b067bd180638ed489f380c390042424e81a5465ce05ff1c0389f1964aa5d1d130327082e1c3005e24dd9948489a1568571ce8085250f0b805570bb64f8462020bb2e0501fbf2c1165aad428dde292141cf547d799264cd04fe445aced6d2fc27c13b292fd5bf923c28341e5ae01780e3e3bd5e62335a498bf788e7f3fd5dc1024e4d651442f6e4a702beb9bad7b4549d82c990ccdf01e343c8275a41984faa6a4506d72255b5d08c41c2c8879345f58917b16ee6ab7375c77dfbe86b2a1fce2e46b2980aa79eaa5e95b121f563d97235d684707fdfbcdcd8430c992d912926cdbd89682881c480aa610769b63d6ac968937cfdb2ec975855e4856f2d58dd50c738aa892f4b5e4319b25e1c5205788f1086b014a7f09c6b115289dc80c3266698f699a903c807ebb63dde62d2367f9b7e03c258df599c7443c22ad62e3c34f12bb5a5b420e02f1def376574f24f44292d91f760dbfb1e87213f831ba9740b3bb47ad14e027168df5194bfd1582d7644fde08e34615ed3925dda568e36d53d844e8786e40b079cb270bf985211a1e5f082d717fa5bcc50959759f5044a4a9c66529809ff14936789ad99ea0746c24aaf118e31f8be9ef4ddfe6703ebe21776d4305df459265f0219750f759ba0ff996394d03366a5b0d42497987ded00e4a31dc748a7cb0ab29673b09db56251f7cc1b2259b034daff14c95b4df5bf97a06b96cce11f81280f61d7ba038b556494b0a4e2814129a99aa98691e8e07cda82c0fc6b2d36b20104301f4fb46cb0a08cdb171fcd2fc0c2fe5caf0c07597aed50001b99b36817', '1a47ba7698ccf3e206305e812589b2610f189f6973f1cfda7ed38732357862702707bfbe7f01d9640b277c0f860605523c7a00e5fcdc3cc8ffc2472d7e0a03f43048c6c74982f1474bfa8b07ec7517a103465c76e12139294cd9472a84e43ce6166066a17828b3dd75acf7b5905c9932125ea87c94ff5be8e208d061f7c655f2028232d718004549c2ef5cf75b1b30e0abbca8cdb928b649061bde38379a50df1c59e36a87b920835752a8e012a22d2b0c400086a6b075783bfd8f97798281e718ac1ce69c937e29dc82bd8ef1ea74d524849a2a1ccac5704b91d8c09186f7791bf53465286185542c3e32ee3c1c7e4507bb29d64c7a4a84faf00abbf222fd2b1f799b9ec47dc21b89a9bbdeb421c24403b134f30f958a23621dcb9a6867c21e187b1872975c4785dd98db0bf27f70e6ef58ba28499c21dc974d52b0b246d32226418fefa1a776bc4be636787c38780b21378432d130bdefb0c238bd4897f563275231de367c6c6f21ae060f82b04045470d3ba1faedb7390fef48c4a50a6f0e13a6f2d80fcc82eaf8de78014d7fdbec53f27ab2727ba143c13beb1962fc68ab20eab89e4aa0497f467eb1e91265781358168650bdad9707ef24c93c3601777c290aa7edcc7efeb4a0838a58a5b57d355ba2ae3fb8255a43de70db8e9b44b19a1113091b8ff19d63df9b2ffe7093827e70d8119917699c92de86efc49ce683500335c0cb72f3117d8de0bd1e7c552b179beba2ad8df731d9f4cc24f293748f880322579f2b23f686734ea78d15142e40e44182a5580fa60a29c8b179ca3647371e5f368382eee12d844424bcda871a1d05ce377da1b6c7a5a2d91987848ae1990aa15da40afcbbae6ab4fcf63157f322ba3520525e4106a5224235f188712f231281bdda561fc4fd23510c1aa481573fdcd3a8d5b30d77e787f5f3fdda3ace9b2ff52265e38bf7b2baf9f5ae2be9c99253c7679d05d560d0892a077e7a161e3909b37a8bf9932d69d39b8a3ae79a75b78898aefcafdd2948dd8e3cb295d28ae414794dd3002ff7f44b0440ed3b5eb7adaddba1486f23ebca858d16fdebcb408e0a900db84e6fec8bf07fb7570e62bd156ece224fe4c286531f85678bc0353b3f137af1983f3b1228104c0abef35ebdf78f2b1a3fc44a4c619ba77616476f6d082bdce296b007c8b718ea4ecd291a67abb434b6047b261f2c2e6de5242202e90f15a31abee14381509ce09d942d45831c72b5cb1e77b507f42f4425037564b6ee0cd38f6b72aae3a7c111786b0d9dba34d5fe79c1ca4a1bc3ccf4ea62dfd06af01abbd421f005f302c6841d802f684e7a3b1cf25d5e32e55b7da41dfb29e2605f0580a8186b93e96ce4f699f225d565812c878bfce9c37c6728b4f20d73ffd02e21c78ad9814bffb2c2d654ccf03bc2ffb800b294b6f90fdf46e31841a34b8b550d99c887f6ab75e6824838f731102572ce6a4922dd1d31f456c0dc54004e919c0e414d7c6e4fca7598dfdab68a74366d3db84fd4a1a2f573a7bbdace7dcadba6028da3d9f987e5b253c6c97962c72c2ef24986a0d9fb1744af9fd880c565603d1b736e8b2a914b0099fc6a89b65faa0248720c632a1ac1b92a29c81e3dc89067212f2dc28edb4b28826b8e2600ebe793ea1341338e0430343fc2d992432510d306c3e0ae54e56f8dade01bfd7bd97b1916f08e34042e03d4692d907aa61b7c9821473aafcd2c81facbd263926ae95a5509d99f85ae990b64f8f3d1c6cb60fb3b2d3576951a4365639a1887b96669fdcd5684fdfd3d56ce135618a497f536c5b42d91b57bffe12805308929f658d219629d098550f1a8349c2d915b227d131ad626d6332f060d7979561a02ca5c410e793bc0a131fcc009b23c88756191d40546255a6082d7f9910edc8a3d7e620c329b7682aa79640c5b549219f467c606a12e2b3a099e5753ffc9782ce47b6e078992304ec067b9ca3b988fb80b75a3397d5c', '0eebd4cd6f4a0dbd0b0c99ef62cfb2eb13a3c09894039a06b5b102399091174b2a966faa66501987f95fd0e225b36bd2fddde5809f186ac7dddaf1dacef4beca09bfffa2465dacf5663fe439541e6c0a7b0c55b36e165b064f998aafa7f4292c19edd2fa45991415b6066910df4341950996ca5fbd8dead3b8a8f1e685aedeac20574af8da3e057e056e3bdbb8606c4668417d246d73eeae8bf63889e27e44d50eee11c8609917a5f6e707eddfd788a304ac2d603447a8f93a361966ad5c6a96288ce10b6800920839aa8491e2dd88901a3c6b77e1eb0532688a685bd0180dac25873b5b9d1b4c790b81a9a0737223b927f55fa9797dda49f202054dfc995d792dbb6f7eab4e8e0525a93b98d2b19ccca79b4323c79b77a2407661f10086e38119a3a9fc61f1fe2848067f1421762d170ef056c2a07a26a2cd40d8599ffb71410e5e74f6a0558f41f6eb4698b85229a90728df8130722a9507c463788e24317607647a12b502b74d686aa3ac457454e63916228a4b7daf2dc1099ff0a1550ae31b692e35bfebaddab2a547ce1f3cb30407924b9b13f64a790fd8cc7c4f73cf4c23bfecdf74fbeb3fdf00773b20a1bfe651a32dc598bef906f24ea0074375257b26e0e37813df75dd512a3cce96e40ee496026b86f20ca37ac2e1643a87b81afb0e2cdc97cb1ea6db023dd8f1cf1c84152880c807f7d65556554c52143bcce971213138767d7201ad221bfc93a44921783a78c6fa2d40e22c8da6a357ab5e552c0493c92fb665efd5ad6522db513252ae2648140686a7cc1348669450c3fe87542bf4aacf6ef1d5fd0a8960d34410f5f057ea19a88e160f84b8731e5855c011f21f11580685c5cda5b8124b295a52499c5de8365721571ba27fe152c53ac60515016ef8a4433bc2ec5bebab591f073156cdee7652c2199703e0cdb9adb4ac289c185a2d1fb9c46869f6f970257dd4ca4ade636caffcabaa88be9d26a43fc5e6db062c1c824c678795fbff0189dd7b55ef513319a7226a02bd5c7fb122b444af451c02518c3de446246cbc4921f7c1742c9075fd0e25f5ac57ef9feac69978bfa3092ddeb6533f7e946c1ef7e54eea592b2bbc1b316d6579fad0a06a98e42fb0020af05582d5c4f4cd76f9e623064f06b19deb39bc054bd68a5ec273ce2038bbcc29b0b4eeba58f5a817b6723b60f95f9942d1855ff8ae6e98a40ac6ba7988ff2e0ed39c732d6102dc59283b61e95959791254827c13a509acf32a9b4c96c62f392c3864a4d4e8eaa37023d1a377c8f10880b8d78a0fd7b6af4a678e07a835cff50a10564f553178becfcbe02d6f8f3c0007a3c3bf31e33ed4e4828ca2a70ec0321b7d473a34312929c679e945ba62b31e0580f5845e8eb30ccbdb85649fa362d22c774f20cabbf286ddd3cda83c089b96ab571f4ee048a0b83ad0b44b7db28a4500bfacae2be6d38a05fb0c690b5bfe565d1bb9c6cc79cc2b966a16a1f2a913a90ce1e74b0f59f1cf3474c9613cfeab4906f6cad2e7411b3923282dfcda972e82101697fb0a649f8571ca2c6bcb61892359758b74c0cb33f6f045e8357a7f10dc0d35128b8e47cc401708469dac81c6ef0e1cc5d1e86bd64ed8a10b7ba19c1cdb2cab634b5c9453f15e3ec714f5aa986cd5ce4ce3d22a75a5527e833802cd23b323677b466f3b89de141f98357733a0c5b1bc5940e60a0471f091e26e48215c32104c265ee8dda21fbc9e29d96877676c0dfe8c8b0944d8db6880c2351211d16902318d1fa1dda338b4aa9b2144519a4d6647cd687b57f473284f0afcfbdb09e82cd31253569aefcef6d13f64dece07fc27f4823f931d7e6d574e5329421260c204c59fb19811a4481942f5d3e1c98e81c65359c4fbe32ad4a40f2b62c14f5ca512e283a38f88117418fc632052904e4d1cd283090377ef6dc2f39f5e56eb0b8514c4b0633ef4d2e6664a66ee6bc4047c0ebc64185b8e3743e31fb9b9800d230b', '29c5e092b10699a51e367b80a33f95c9b8f06b78086cb0162ee10b4853c4a5e813208b17f592dc2f8258e6f9c2c3d5c2aab27df96626887650729067abf4cd142721cd818abf62ba340488f1edc06c6496b69495f2301f7cf54a9f9dd918d7c915258f5eb1913ca88b4d212055e933d11f1fabe472402054bce90d3f284f0ae6026416ab8d2f15c0099d5839cc62a6bdcea672a5f9e7d76b31fe1cbf88a0ade616f2f6436e5014a3333c009d47c3001e40001da77efa9566038c8fa835aa97532a7137915f4b7d67dc4ea26f0684e735172041135fdc9f4903fe97f5e26f258c26697347ed3363743c50887778d33d8d1ae3752ae1387b445d2c1e45863e1a2d1f5f914c8b571267d1b64998ee46361d6947d86a81e3f9cc06cfbb9828fbe699243a6c9668f23ecb1f61cdfa10ff64524e76d7627bcf26178328a402b299081307a338cc9dca8beb2ca827002fab209b561780ef69eed98deb3bdaaaaeb0431117e46435c98a85ef1a1c7b6791aacb9136086502a05df23ac0bdecb553c9c26414b9da2713ef213c875c33e348ecb3cd1f3d0fcc4f4e7e4e8d88a3263d7068332c12faa1a291d139b8124f040bff3f574bfecd71b51a9c3cfff8ee3e922a43c92f20950c2c3d43d61963a7a76decc7e69351b2da45cd22392494ffc91e5dd39e164da0f77cf9ed9a6c0565df5c95cd326740c2afbfb2a8bf6ba5c7c9bea5fea720aac70ad4a7a386f72f293df2f5452652e1ff8cb8ccde04a9a70311539686b8159a279654761051d2f202da1779f5efdc8020d544340f6831087acad72e9e920341f955de1d8eaf5eb411f450a084a796927a4455ee24c893e2327c686810ff25363cc6b5fe3dec23304f84366653cf2949ebbbeac1f1af48b21b4e61e694c20e6e6c532a1caa374fc2522d6716d2e9385f9139de2fcc629b0d2d56351752a02a5908d70b583c1183646f2bf516605bee516d6e2c815709ce47f7333b1ec5ba2ac25bc32b0f531da04390d1d006ba3d083dcd8b61c62d705fb56d9cab433ad20bdd2d5c20ddc8c8e130ac154ecdce855b4b81af1805debef58c4782d68990eb28ef338bfe3fc7fb94fc8764f467acb9a1d84f06f3b81fcc6c5300e8044776eb292dcf522c5ec78282249119715fef319af26572456bf37d83565aed4109a6142d17d9c17fa574dce964fb1fc94a0571543efa49aaa4332a53f6314d37df422e03b9183b1f5efa673e200eabce9e41e1ff4f1f799de67c04e0e6f7a6508074f51e3e6c41dbf2dc4d18755b7c0188f01a4d45f0ccf764843c786ff7093b216b72192142f2f43e4aee5b9232a7efa40456e5677a8bc060a58b319a1dff35c285961dc44beca1490c678245926bd81bd008e8412305da8877368ff6fd47c614ebe01404965ab9f6faa6b5e29ad2fcfa407d08f20b4ec58e13a56ad1e36d5e44cc9102e297806f917c5be7ed6770f660580a021c902465b01832d59a2f6a8563356c15f85b1ff48d826d7b23fcdb7fc85e557c46846ce08fea95dc784adfd67ee91c176f7ecdc1f20846845fc077eb58c79b62cce4bed9eb8e70e09a425f0928f9d723cd019b01364ad6b73ea88ab7015003d12c2ce5a5556b6563b9eaaec0b23909173c3f20e929dc314c8a0f718a7a28e567b7f9d0df21dfc30cf4032334d69a4d06721a70ec12feea24af323a0b685039b01fc3a3064acf2deb096534c38c166f2b3107dac97132adff93398cfbd12427fce23f295414fd85270f8eac636314180395969c73b1a858d5a3c4cd80ad6e6434ea43a4632ea91cf1b7bbdc4733f8221cf023800619b2f4c2b8971d428d57467e41283f297f70f7964b6613a1c2d4782d80d0c9298decf2c8231cad353fe6e6c438d17f9a870ee22a83f1987ae00b7b1ed0f4ff7c57281b1aea92497486b0a14b6b2042645212d72f25a6dfa527d9721cfeb0ba00072574ebef5f61d02a372dc08f1b4205f70efa951d61152fa14dca', '2717c430f43b6f9fa464b72dc401a7b51bdd40e70203364b71f7c19ec3fcdd6c17e1644fcd13aa74b384dfa540c9e379422abac440d6ca8c60f9fbebb2ccd63007bfd9586c011cc23c3ef374fcc2169c60f35b0f736b2acb3f324415a7c7d21c2f9ba816e66391256de44dd332c10f9ea90a42a1d32da8c440f44ffa8c8b4a470b863c4003c1bbe472bd904c9c61b97a931ce0b48bf2580263e61f4f72d731ad190731c80423f47084c7eccef0d5734bc013d95ca96af3e58d390c8c10008356042c283619702718c409ddd57c35f8943dcc39a9ffd10d4eec7899f835a7fac80a6090942467e712b71306fd2be0dcc78d1fd9c70cb11c41c664197bb4e01db8233ba01c9af2091998853cc856a9c8add178d7ceccc83967585d7f45e1f377e61798e95802b55dde8ddcb359b9e2eda3bc1ab5426a970bf328914873f822f3ec135c588e3435b4c455fb40a2839f226692f747fdde8862cc44a01b8f4d5c1e8d1e400ddf2b02f725dd6ae469a2688bcd65a5d6284c52310f4c9d32aa1c209a14257eb42fca13ead86275a38e8bb79dcc319e2bc5911e5baa5710f6660cd422992acff076f6c1e55f73e36bb7cd1cf037a992ba2c81d45d52cde4e5ed9162492c0eef50bb2a652d1726d337daacc6f199002b24a1c4a56474ae3a917a8232861c0baf70d2c487a048c42bdb9ddd04335ef80cad436db792983fb9eeb8ee88df161e7cb57cde64a227a33f4f499c0117a53c0b7213d983a293caaafd88e83fb5ad163bd992a29e5d929ff6370b3cd4a4e852ffbc5e47e4eea4fd53ae0d29f4d3432e86b7ea9fb7cddeae5c1d8a15ddad03b40fb5d484c6633aefda5999c175d13503d2455a3564d3fbc2135c90eb67b287080f80db72bc922d1ec7f1036e097eff2626b2a1aed6a1b8c4bf516077a756b14d2acaa2c60a39f2fb9aa7f45156e3bf088d59fd7edfeda07e97ba1686c5b5dfdf89cc8a37b430b8f22289ec38e6b2cd0afc4b7acd74fab5ca2cdd2ba955dbe86d9d73b07770d7aab408dea2b6c780b202a27413d739fdef6efbae5ebfeedf3460e94c9d50a71898731616a1b7f6605f18424cfcbe2633604d846e9cfba9f461db382894cac2a5c6ae02d0f6c35d01e70da25809e96f08697ca4e937d775572dfbdeace960c480a20da5df39c9fac45d2d273ef534e8e00afb6be9d783c258bac9d022818880ffacb349029faeb9dc04133415fedb26c926f47738fe73f68f7009de83c1d9b20beea2b3d8023d06911b2fce8d628c1b7c316b8f447c57975a9408861e81c2d4aaadc85cca3fa27c248e04cc4d9b73581f4e0d7391084a71c179921ebd889bdf0498b0faca661b07a2d71da9014a6d68cb1de4628c38f6d0a14e08725d0176482520500bdad97618d2b31af6bd2a9f6c0ffef7d6944ed89854015317a944da55c904165420074c337a1107ad031cd63cfe6f7d57c3ac4b7e0d4972dbe202ac657c8c76ff11ff6f812d6228ed2550063905dc955d8c4d3d5770dd9f5a4c727e8789c4070a5a4ed1bb01c6142883c26985fe440f04d6563d1e6c6ec21dd9a49772291823bf569d260e33031d0a6e276ec44172f144efda2b89bc9868c996c03a6146e482921bd3c882affd287e3ca609e8432578b38fc6a113a1d9c23ef74877ee2ff4b946fd52e60125ed0d70f6af74c3014a27c7c6e02a244cf67d1940824bcc8b0546838fbc91dfbbcb2d5f816e7e3f4c300d752e5a7f9e924ef27083b6a6593d580d4aaeb0483e185d0245edf7fef4392ed27884c18ca50378b3a608fd82e74a81288ad6b8f4a62e45231181bc2cd2bf75d90d09dbdb2e41a78e87d5e1150b50261f6011aa89405c7507fb0d3de5f87ff4cf27051e251218a15af38d3ca6529e407eb5d2c8881b201714006b76de72fcf1be894cefc62dad72b127246073ee203af6440523bb899e090b3e4e57b2cc69ba8e438482a9db55146ff5b6c7d023f36079a9b02b5aead0d8', '01408908571de25b112584fba6092c0191b5943f3b0bcd07bd273426704b7ec61226c4311ba38146a55624f9a858ebddf2da2ebce0f31e902fe66dcfc821125e2fbede706deed01c3762b4e3863d28580d253e2a3657f151e1acfdb415f1b0712d99215435e5eea347e9bca2f9e67329478ab615a501cb0028d3ec36d0b9324806a8e2cb9e1b36ceb094fd28fcdcf00589d6eae2388a69622e5661a7af5e79df2eef9c84433559abf0801178ccc006f6407a78409e18f6eabe0dac22985c493908db4625c11d03845a8d4a8c7efef58acee867cefc24637cd7a1dc8468a0fe5b115712ff0952d63f3942e553ae5b492e81b0551b4a62c292f320d057b8a8982d190b052e6b5713a8bc7e5f40f78b35c8b272cdda45f0b0b9f02143af8227010d1f02f1020febf140918781f7f6a93e1bf44ac42fbc7594b2949290c758887ffb124138d1155fb8ec08c0b83a158ce34f6bdc34493a1551ce1bb7d99c50cd3ba61d914e77b568c2ddd47384b1fa5700041aa8a82cbb3deb7f47a73b376565277c19df23d4f871d4b110dbfb27329ae70d49db4151bc4033a080fcf486faff227c07cf8a616c28d25edbb6cbdfa0c237847a5a0c2895e90554a3d3c5c18f77d8172ee981c227bad011d61f8ee20ca489ef08e3ddaf67a981ac11162c4d184f9fb903d09d01fe49156a3a33d86994dab086badeacaa889ea948a669bc737f648a961520c4e3992f54cfc05a7992f20e39e2e8d38d3f6bdd2e8e0e360788ca582982133bb3bcb7f4cf675e429d2227634074880c9c34ae6ef2db5380c52fdbff173613ac8fd52d738ac68852d56d264f80a903e7543d60fcddf0741fca8ca6d9fcda05afbb2acede4097845e32fdb04b320b7da3a8cdcd632470282883e303bae96d1cf09ab1437b26cac6bbf152a864b91981591b31b988f039538357c3404fab441ac89c39cf0bf432f10271f1fb20576c5b42531a84121aa50f85b535d3f0deff024e285d5b67f69d55db1a7a4fb9d3be4b459598bbcc262b4d37c9722890e7410bab52e04abed11bfd8c3357f454612fdc4a0de57f86ebcef6f0fe2a44aa57321d93d217eadd2bc0299f3cbaeb081a0caf1f3c0b58002dd19482c2c2f6d5694b0072a24cfcfeab7a74bbd1ef620ca161a2c4aa21ae4c3b30792c5cf16418cd260880ca15f77966bbc5bdff62d752223328a69b97b5b257e6bdcd0419fb012a23081400b2282ba552de89131b1813779692f8933b33ce975fe34f7ed45554ec3e259be3233eb0e2daabcd94718aec79786b226809d413ffc4f9e13c1cd1225b900a4e2bb0a604e9067dfb6ce0373d6d5cdb63426c1e60a44f5555989670dea7472dfb6954ca1b53ecc4f519f4c4ef1da0dc803f70797ff5374b19881fdd76a8d410abf59f9c38b06c7581fd82b6ed599266f08825956d720d5d3a27e45a30d5fd2b0ca831244288d041be97f2390bdd0618275ee015ade35f6f97272e29762fdd04245d864e5e44a67004a81a17a13e62986e6813b3b341c9cbeba0f35ff5a45916ede1af18a81a5e3b49483a5b337a5567b8a840a269e07d397f50eef330a5731858389eed59499a6e1b1ec3ece59afff7d82a8acc9655daac5c82e4a32a4806115952472ceb1831e451c097c4ba20184c3855a85f4264ae108429a175980b8c0c5729c64eb732d5fdae0014d9ee8e117df69d9497816bf5aad0cd36fbddab3a0830ea2efaa52a3f8fa0a6d141388fda52d5457ff5af03c8f70e6071018af3202cdc101be9ff9bc566fce5d746a299ab9c916cef19689f49367e530bcc5869ad1aeb6734d00b2f10a5803b152f31de3a6aa67f9af49d95d06ce33768f8d993e00cdd4d62596ccdea81612e46ea071a6946d2d41803f72d2bfcf20c57b3eaf0b4269dfcf0360d5c39347fd318b85011604355ef6c27d0b4d9c3b0f3e991b1d36203b5ee7f2f86c8641881751485f4ceceec79bd8e0ef6be16732d886d5e5875a1', '2696ca3f016a2ed11ce0a880be7acb547ca580141dfceacc304a6913396dee4e0f57cb4ce9dc243b463c1e83bacccb144bb3011a739addff34b53a8a9a5e9ed116610fe3bb1d23d65ec0c654dc837d13ab5d014d2869c96a56bd2a5abb00867d13846b24c0c8217e05e482311b6cfc09147f4bc2b5dbf36664b84c6a4dc4117116a739bc6bb99fe42357f6a6fc4ec0b4b8b370be6deeb339ad22196a208a41b8112cf80672e798611e206978916ffea0dade0446f575d06bd49390641f0b5673268228611e1bb299a2fdd29b6ef8d60bccf52f7a8da4a22a9811b9f30919495d01571aed3f746f6ee1abebd7ee696f3adf31b6301ace9dcc89f7dbd120af6bd60a320bdd95caa169774d0e5f10b4e7248033f30353c863ca4d8da25b4f7be4de20fe36f75cb4dc151fdf0f3b71a19f5ae9c7c695d8acf47fd65b0c942e8291660d597fc38f06e9cf5d3a0898c436c59592345a622572a1d119c7f30520bf58f70264b1ce813e12bd07d503f90714d9c2971a82f04f7ba9697cad9b9d020af8fd0a4f1d9f1f3bc0f59b6dcc33ef1c0e9c3bd69df424a45eeb25b3c360f4b001602b5fd9b50260a13863fe68f5c6b621669c2ef31777c2055e1e3c0c45501c49112710b4db0c47c21efc2d64533f414e00298f9afe7d06d43103320a1387d599f40a2751e33e075fed4869a6ba6d730d355bbb6e8ccee547e5e940a45cba6e97c1017d4b00c7586764182998127d8f1f436898615a36dfea29733d4c17eca027721406afcd3a0e61e6d2105f6543022ea1f7956daf3d7daef9dadbaa1f5f4ae4531706f5ab3d50eb757a7a6fee4cecffac0b7b319eec15ce1f819dc6beaa655e6a03051a5745e54d82950fb1bf9db1a83d1b00dc46106ad1bcffc627b5e4c8978516af8caaa48733f082fc16c3f4d54fcf80f4bcf05a0012e154633052a0d0b22c0f06c8500a319e3717f03703e5a832bd89f8a407bdcdb82f3ce691fdb7b212ab0cea958ad021aead6cebcdcecd7151f6fad18f7da4306bd5f76fa1fc9316775827b0bae6a0d0cba9f7ffe269a53d5897b0a428d8de7fc02f316d50567a22466c2d6e7282cf7a41143d0326aa7210ea3ac0e8357445808e858289a22fb99e81bd0545137bf01a09a97238a5b9fc1978b0e4968f2f722a2d5cbb85969811c97ccf23abd3309d58b40120415e6cf02d1f8ee39f813d7cd31391a61643dbbf81f5560c4ce86e1d8e4d126e59300ea64e59ea42c7428b116d2faa24e7c3d43da4bde507b47dec61e7c4304c08127693cf233db466de779c8296215e1f94611f4b4da225d19006a2301642c1a55048725ba7233d74a7e56faad8b1863c438eae219c3d29629f2f237e425585b88bd743b881fb7c1006c7fc1c3312cf0532282d3634a203f13503261681be4345237f1b2f393d5ac2d2587048e9d9d2cd85cd67fb9473037d63615356845851b3548f81a0127143cbc721d7f9702c4d0bbecfab4946e31babf6c3ea40c4b14c77415228d2d7ace9505568243ed1a0572e7bff77caf6e428ea2daba22471a4cf2cac84e45f0830e0d4d6806cbbfdef7ef83df955c027862abb4685ab3c0d4a0888e932058299b572c04baf222ce567bea6816dafa5d1480f826c6afe86f3876e234488296010d982398cf58af9cf1ebae94b078dbae632091fd4ed2d4f0841d9d87823aeb781242bf8b706785d830094ebe0a39be0caaf2ef6a021a9a9feb194206079e5c895c01ca841678acc35d8aa16541b9c5fcfcb010fab5ed318bb3a7c39b14a4285d34451250d16710bd57980f8443d8e7a37200416e2e99a2136084885b86c776f70af09eae56df60833b982fed37b5c15fe22109592c8aa7044547e75d8819258a7cc256c7f114784c09802f3f6a4759af6582d6cbeb82ea1bf68f3d3f1cbd6fd265f324807d08d12c55b18609e2ed9c4c24c0121a604e864f3a9bc5047d7b416514b9cf31511c3a7e9143ac32dd05f4aec39', '0a67b76f1b3b7c65600a00631941a08b7b00440dab457e730de3f6f14a4c0e0d302722efc91fa1a0e60518398fdd55998a9ee1cb2a6f33b20f524cf3598ba584271affe40f5a787f96bcaf7e06325ec8a0e885e36824a8ed2911f1008f2bb909286f278309da3b1d9c6fde67791066222f033d238ae4b57271f7b51c233bfe100bed67cdea18caea5642fa10f3088e8e13c9f8c87c1940b51a36fd7bd446dc742c60e6aa6937dbc0c8fc5ab90506b84aa3aaae68741e9ab1b38b6b4de9d728ce2a9fef9366460e5bd901fc79988b713c48fd54f820596fe1eb4a38ccf768c08e1cb6f7786a5e97ab0ac72166d48dcb95e23f5c06ca16c64e6350cc61bbe26e0d2eb0f19c4e9f76f962271d3347061b9cde54580ebc8200c312720fc6cb2212dc00f9a2daf4d31de7ceac0d5db3f34dd46fe7eb8da62c9c34cbefce35af218d80095abc2f1fb8306956da73e1ca77001190e8b35e764c2118c1e3f7cc63baca9a0d4e1f70ceee94a1351caec5b38a3fa788df2825a96cc3a7284d6d089fbceff81c63ff1dc2cd1271c3c9464b384c23ca5198a3f3af8817d15df6e0becabc35660db0d320ca03140dd52942add817f7394cadc0ef42b4e9d6f8b8cd1a3f1bf44d2b49265b4bfa16f0498a6b0706fc29821315bd9c3ec1b005434294884d50127c16689ef93e379d88d3e3ed7da60dfce78866f4aca701521c3c90fc106191b908227b374f04ab9508856c05cce72b55588a6fb72af69d3975a05730e60ff34076133a28e90029812b674f2d87bef31b550dc2bc4b3cd4782d242665e1580148491bd8fc9da24bb4a6b73d9650c1e152ea4278c927a808fd4d89ba337649ba0ef7187845dc5711e6a07a414ebeb6c9ae3357c65e0c7075434ad04e73fb56a961f803cc0fdd7a37194bea129beceaf2387fbff6b73265f18d056ce74dab6891dfbb01607bb158ad75b15e2c21b47726534195636f9c0bf440ecac8fd69e26becc4119cd5ba461b2ba56832fc4a29f1861f88dcad9d37870b563f4157efab3833e7c20a3390391bd90714f85931c40e154d43084f7b149ba77f0f56a8647625246a628dea94c75bf70201a05cb7e7e5a91494b00e472caf4b814e287ed479452f6b4214d6bf5c048637db39aab2c941bce485e3bad555e479e750cfb4671093a72461740af8e74d8b5f3afeb25edec0bd8c094e4b648c14326a28c2a9fc936d444c41f7a192826dbc2e3d25e829fa844390a90fca359c1b3e397f6bda2ef776576411b2884b50c807133e8f1d4d93a13b7d9409ec8c6571a58bec062ca247b11d8e618c8ab2b523487158f589202b27b8a8b044a9731c81b0f7da968b7425010173a27d0a4ec42a9da34a815beb2d4c1b0d9cef5770a50606f4e637851134afa07d9190172dcbbfbe2651fdf6e0c800d96c38cd3e1bb36235a41c651fed8baaeeb2f089f883ab5d98a831406c038c43d1f96ae04b8672ce797039072d9470f3c3efd28482f5874f44d41d4c76e20048ea0203d4874fb8d06fb434b5cf0328fcd8ea12bc63eb24590a234800af842a204a32001fc6b7323aa5a451c43b9817c47d73013677921ec761ac51627b9381e6556583ce17ba886df0f797148fa4e482c3582240d4c55a0b562827f9b1921bf8492d3043380bc0104abc3f4923213b6f77f371ff619bd3dff29e3ca449ace7243c24b34452cc65e30f8ac7b955b847cceadfc0a21be80a79b08d94c1543403e4cec0f2da1a0b9234ac738d388b152751da08f14c06f66d42c4b1d6d3912595f117399a6839dd0484040a950890645de4bd8531c9ce274922a6ed967aa7214266e40f615a4bbde95e7ac55dab11126b86b6d8503d1a836a08b7b4c8c223d0eb31123990eab9c469238c3f518075d296ea12d462e21a1eafabaf99573e3e71e05ac71cc53b4863d3ad3c5d335b3f31e964944af1724e320ece2280a00032a75703e567cab40d3109e9317c9b0831554b0b7ebb0', '13d23e7c9457ba132b47ce890ef6bdd9c1e5ec17e8c7781a9b340731721d35dd26b2aa030bc3a0c5b54304cc9b507c64f723518d0f35f833b94511a075d00f2c274c5cade3980160f59e55cb220c4ef737fbd647a1f980d6dea457abd0d8687f1487bd30fef8586cbde6107816c9e7df39bb18f65cac815dbc5277071dee85d5059dc7d66c7054a99a344eb07963f27142fd67874f7305cbe06d3ecad94da4713022066c0a2ffabc1beb972738439c21da22ea280503538347c317a2042e7250069046b56201d2a9b3f83e7497b4b980a6507982d24f06b45073c14a4900f9bf21941a5f6b316ba7d7cbbcd7ffe9a16b866743586d08761b996886c8b66eef3e167ac836ad4c6e5f77955e5ec5151614b8168f7b9afff089b272a3b687ef33c425a8864c8e4819d50c414be77fd81e3a08dc47c7e7e41e79c73b41c9886c702406d3505767e6bab59a0664a9b48c13030829709ff939ea71b0b8641ad447773f0d7e486b33f58673513fa289ad66e0fd58bbd735a05f47982e0518b5ef3af1bc102be34c43d7e1860faa1a575652187a6e6bdd91cd5330a0b9de543e5725ff48200abefd18b69b9929edcfe82d0790a6138c8a1a70760853f82aa53eab68beba2a42a20360636e4b38fb1146d087f60e82accdbba942279c793337ca29496be719ac2911d04951fb0be9d9f7b63c646087cec34262e0f30215a8e8cb61c9a03904842f34baec13391c35d8d5cb41624efee089dd501d316d00465ee726a4fc711d5e8f26555ff3083ccf7b351fd68e700e68e2339de2434d6460070152a568cc00302ac377148995e0da6bcd85ad00256b793ecc13c4209260b74e71809a0e92227a06ebe04774f5b114df024375c34a292800939fbb23b7452040e54dcb9dc815b74a2dd3e90119da5cc6e17488e25d192b73675e6fd66dc6d354a673e7492906d2b388394d3d9265892a44a36f1a5f6d273d655ca7712eb92b75869396f8841718892034822c65ddabc5f221492aeeec4446471057b6c67331fc607a03069702d0bf8cad2120ecb343e5453fd1c3e44ef6d7023c6825952f4f206dfa011e3905b5cd431decac51e123921dea292b8cc1da034782c9de2cadd67d8733c6b13a03ccea6afa96eef8b0f05d8d6c87735aa2a6e57df89200cb6dde47227ba229ee07f0ad0452f885e0996f5ae33c949664d17f3c1bcb68685d8897c844ecc372ad1d7d6d61437411b18a9b3ec2b7b624134005a8772d1b40a8ad7f1db0303697ec1643b00bb06fe55b2c2400f8d9bd4757b6a77d8ac311861dca11c841357cf8de1e61a22267e6dd1298219d1f9cd3ec347a13aceeefdf0f5eac3865e572e6cc0c272d4ca4919fef9c9347ef94cb87a4153724544c5e7086d02ba5a3b423aa9c521c368d2edfe602e4d1950e271ec0bc813193f84ece55f99114539bc8eeb1c9d901eeea1f7271c5abbdf79539d4151da842b6ad419f4c1f1608eeb69de0f91b5b06299b04dd32e0a4539dbc7c66c3375038da1d4f8ea3fbf9614c2e95e816b2572244116a3cdb31b6ac1ba43441f67ac639c6380addeb7af60d9afe211dded5da2ffec367b785ef7fe329180723e82fee628e85015562c3c733114ddd7f52d20d1b82103715f537f4d18a60493ba2b15b7ef166f7d3ba0d407b301fca50486c831bd3e26d3432fc01f5815182984f318ddad1627fe16aff7ea390914498c3031622cdd422ca087b284c3abd2a4948f76bb88a4cdbf5338e2de03581475f8c8b752453b1edddbcf560c1d82a9040129125313758bd877bd87efdbd80d2c3cf6ae92f434e4ea1775057ea989de749b06b77dd7deca35134e4586141abd00457c77000484007ff62d36f77ce74d05244ec75276f95ee5640b957b7922df88d5cb81b1a3927fa05a7108be978b28c7881bbe81c16162beeadd374defd52ddeb61383f24257dc685d0ea6359c370b97f57e1b0356f85643f3e09704f79b95e1119699f', '122b894f026b22d00110ecd00e4719ec5581df04b0012a498c27f96635f23a1f00959eac69a43cf5c07891211dda11ccae8316f1850f79c724448f7f1cac7d4d263cc94b2665747e97c13f5f47288ac0dba2b44143d5c7634767618745d22477226001f9d18f23abb38b62847a03b547203a261c256de08085795e4a376dead8227422f320406ce57a95846c7f0bf411e6b139254511b62048205aaa2e19a07211a9156a8c85fc5294ea25c270ece4917984ed10ecfd4a069106172b5c35c11f0172840ed94fefd09109464bee61a145de6d88822f3c3acbb20e93a3c88a3ab529db44d146e6d38a14d5f346f8ccd1411aa558257c10af12e026be176734f5730335fced0e246b591779236310246e2770931e59a8dc4e409f5623e4c609e664082411a856fefd18eb99e858826cc3efebdda279e7ada93f2a28b58899dea9122c6f63aa3c3388452a2cd2095ba98c20616acd9eed6fa92dfdac1b431d2e1dd42fdd87b0128ef52d0e4563d251fb6631a27ab18e2824dd503ab05b1c1202cfaf27584c28216ba74780c0bfb9b524fc350caf92b030f65ccc7314771b2c69fa6430104f4833257b2da0ff7f4f6912d52b6176fcd92df80fb24d0074fcaccbf7911cfe42189a65d1e4a7603a04f2e17c2651ec6c96ee421b191b9fa65d186a2a5e0912140a2e3034275059d56893a30f386a1dcbc34ef6f691bf631365a9e699a4122fbee179c7b231b455abb0898a0226e1ce1fa23aa9d4e49e542b236670e047283762984c62b917ef9a52c4f2b18beb58d6512515f1e521a130668afba8b7bf0105de2465ad8165e10061f513d247febcb5cee7ad21fce3b726bf40b03743bc0026d36a1618cd323178b1acf85a9976f3237a3d39e071587638307c2aef81391ffd2eb6fd23d8379689e97eea0fc64041361f33a8f103fd5059fc3f85b08892094b3ae2c92ff5ef0e5e6f1b9d0e268ad8894e952668d1bfaf13b654c46cd6bf27c9c93acd0a16d5a3c50d149f7192000747dceb72b41ab0a8b6b90e1540f8e826c8d1bbf838b7af6e5091078f12b66e5d373a4ddcfa99294477aed853bb62ab0a48eff2b64fe23ab0cb1a1ca807b5b4e3ec5f14b3c1030ed0c6f68a9a6677322516f2da08c754ff829f1db7341dd12c768a5594b9b0472e876b835da6d8a225120481e1209ee88940bdc367c7c00ed20564fec85c3782847f5b21f8f653b68e273da697eb8f68aaabe04d52ab2fd8230e41c76e76718ce39759d8eef4e382202efd2deacf683ccb9332a05992514b92cbfbb5fc4b64b5337cbc184d598a516419f51bb15dc87d19f173e7c5b0a0c285d895f92872b9673fadeb0ac2d07ff5d319d4ffca7811330714d4cd3c21535038d1bb6b35926f6d0d89d54c9c0ad6a1d011e63a4157a416f407c08a574863d5492e3e65a7c406a53f8801793690d80ec60c3a3a4c0279a923ed7e7dda63090fd276a6b6c1f84710a613d33a8ed670b6b2103491c82f1ad0b05445743b9c1518e2e508dee90d980dfac98c5a99a5aecf52076a1037626f863e963d4badfcb280b86e214b1b1bf3fcd66a3d65aac8a7799703d99a98c7131b777bd4de909a3fab53618d6fdb3273668e8e61be9061e60abe1c0a24d1c24f1fc45b3b27cfa4be3d2aed635e0331fe1544062803a7af3915262d0e8163749118d6ae8fc10c39110255ca694c4aea2132315a1197e50b9b1673252ae2898fbc490e6cb154fe6fcd583d9933011ac5760a6b022206b15922a0f8160e9dfe666bf06ff0ed978cd48693b8ee60f40443cf2179167d7ef13a7320432a19ec675340416a5d5f90347f40ceb4870a85d8b6e858c0e35e97a8b9da76e3170ce9847282ca2a73cc07ddedf1476fd4fae3d99460e613ec0456ac199a346c21b6dd4af940feb25050c10e6a1f61cb01e498b05acc44aae80c78c8364f7d9116fdbdbc4ef9663f2b1999e3c9c53040a0e71e3d83adde59785dffdb13a7e540', '15fe514c4913cc4b0a681379452c3fba8f3c2c050dd2500de5fba1d1d1fedfc226596f8d2f45a4c5b03b7a02b6f2b782dfd76b2818e49b8b68eb991e13b09d9d2250a25f009db651c16ca09d4d68cc89abba37b8a48c3f339386cf22b4559034185ccf41615cd5e5e65690109d1349bb0a0f8c3f6e63f1f0122e95ccab4207100e1413476585ad4f49e9a6255e467e7af3c4c564353ea03dc5c50a613df2d2881e9e33f6ffebac8d9288a21927e596d6851117fe486031935bdd8fef160f83862276d341bd902b6ea1cf14c27a7ea6809b95761d7611a5aeb28ed6f589803c3f20d9cb66e355a47556cff8bc66d2660fb91be06c83a82cc4aa2f37e046f35f440b747e7bb5ab65904d2001d4ac19529a0f1a9036ee9912b35b14ab174c8fc1d61cef574cb71592c436c86ed9c8c9ff5470b1a7106e6fcfc7d17c46e9d52969e921efc67d288971a827086891aa7f95a043941172d274ffa9cbd25b2b1852db8f19d0f66c3baca2c296944be6d83914388108a99b8e8cacf46f52e5219eeb9c8217129eaaf8d18808f97064698bd326c95ae2047d2d08c3d3a47ef89edc5834eb0cec3df004747a2429f9f8eed695c0dc9c1f3c3423678abcb3af2f24bffd970a143a13c3a4e6ec136557dcc5537bea1e13e1f38fc6dcfca82e1627eafe15b2b91e2046b8de305b8521b2b594dfb511ae7fe132ac114315d083d85485131e6cba1f776436d19ba48fbfe6501cd82ebb9459d04122959782938a43ee8b19ce0da82d7bb8152fb6ed16f77199dc06ec7911e7eedd4d6432f170895535d2f749f7a730364ffa693fa4fb9f7af3d463248015aca1f66c7633a6ceafdf2e4d35bf825f06bb6bfd5abded4397aaf51cfcfdc7e7d2cff1e9de4aad6d2123bcd60cf5b86a210969d7427c49cc1f9678bce747de3ef00139eb3463420e83a7f1a578520b7011aff16646252ed224b722c37e1727b17ba3a136cae10bd17c358f36085489392d741fd7d1bb2dd67c99d40dbefaca07f9a964265af573d3d7184c748712d3772256daf7536e6a90785642c542e6fcec53527f6e4f2259e844b88087ae732c832718821678318d7cfda3f66c58eb58d6bf497485e741f8665132de56a9393088287ffa1c002e3f9c3bb9c103b5253164eff206fad726d6f7fbf05b52dee143522a8e8dcbecb7d9eb933f20e326e018cb09446ebea228c3d26704aa64f8979d390d63f9b3991b322969572820a2e7cad3b62648a11441cd8342a5eaeaa83b393a274860782ba0dce8f8c0f7c8403f9e894164524c487abb1b021ad6fdb88233fe0a6fd0550a7f5a2f5dac38ce3b8d40271b8fa8ccedc190b39b3ab68fb71cb3c30cf2e36716cb45b94757676586dceb5404d4de15e0657638cd51a55e40d72b182d9e7b130f82a3260704ed2a6d0df34cf13f9c5bf6be7c863beb98f22996d0452ff6c6a364ad2051b416c7dfe908daa65404b180061ee34388209e095d495ae91aab7277d6f7525eae6644fe9dd149a46cc9f21990f4b5a5c9f679cf93bf2e742771e02373a4059f63aeaf782ad1826ec85a70e5fd532a0623c535fdd1fe44202334291ac6062f4b6ab631a86567f3938ad96c45e0a9c3010012754a840b3df20058a4ac8113ec0dec2b2e26ace27cbd3f7e2af34ff8e5bf1421fc847f77c9e52673bf0189add6674c9c8eae1ba516933ab71ff429fafc79a142c01fabfff8ce1e0383953abc4b1f1b163ac53478f0ef0775ed7c6f1419e3dfb2dc8117cada762f9f97b470e38659f70340a917553cbdb4ecf8535d5c2e69738ce67c8e65554b01138b2ebfb89f49c2347e88610e1c4f32e827afa6cf3cdf65fc48d369a1087823bb3299de55f634209daa61d6e0af3f4aa56305f92937118cc3eab69c51da8c21d93d371be769d4ff8895a4d8f6815338bb667df911b35db832e4bb090201c11e2eeb445fa065fe1f93183a0ef7d1ffb3d7f3fc19e93fc2e121eaca95ef72b9']}} [2023-09-07 18:14:19,292 INFO][30622][free-basic-rosy-SKALE][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': '', 'publicKeys': ['13186587071177268014884662051300647241243749984999707635551137883599174049477:684064195071186527337856010442245199941216137143443945544046674360427315442:10217196268909022034030854186855466055306045575579577596269861786024692959163:11398854978636717767136577756743180832535017337796605828509639100284668791606', '17824768155940070750998909226513339466402208475694687928469104169648270505143:15754421018389103181182514067221372247423150352597629501165429392565344825311:11194987249617056283272995429092918447920204272849598539279795257720320091468:9458643502245579745631850112576163709762933731488010128238245829383704905588', '7428359847655644297453601604318369557601497634829480666167246507137074932282:18322660120388159319959183654977337089195862941827390459776161188300548012693:16116200119491984690053888238665090216814035142733512477179550611286241574998:13369583904913428260705894254482505851196406622646325871520571099898033887132', '7488414575974377775057108191854192101337043579884916332621909975150370239079:13603590350565681974416943527842416093336963607445743103376284885204287956996:3914799598989059908769786491690854021150755614979336482034112009325861505750:18409979682010110180145428476879538925807154918689524101748452379958476405571', '7995355429030127318418752021973675837718249919773582466719811529394809960191:4528188282924334616501460502330696292623482271065819719846868451276699620726:21748538382302078084102232836258046268143007312674556676496627725455426736090:4454177658685400744659534726576699311938096792097316112187058625645896689590', '9865800907282446428380957886688706147021859631721644107010121064191016766526:19969966416854901652341551526541811964933861886391316637118074817923751071073:18549080483142848619648491437891470794845072085162231576551669635246441325991:3900085175477047706017797202210112531412744596654042111837301286860933095240', '16087141702042604956880970862372919288641041934347102582484190571575172358384:21470592144676932004549641697831220953720029983912576636534666059459751181436:7949478202832393588618706328678882792658949385799403566619097395008551013263:2280671044615127145482846088481580816210177094537080362078876787646035818155', '4458981546990827356093432445349545891406980209102063789815901662819903711413:939828169292943303912196448394902207210454516286061017710278691852586100625:18407410674944689106586903108758602846969083658702476457750890393197824468090:920433794658175687096928835007020881095311081807020560324438768137614421760', '10767379516435907284712084696907880130896285335265064554247328307484650843956:20396072550208804386597254017569917744467271812738647210596227060216652215455:1416398946078687792873606654905090384074154049464894091788451945037372215961:9640756704373228058309874248343807414033030202314280571538522249642404644487', '3109821963528959325001042343362462272664277478850780996880688020952533380373:1681621781942292154153075727572965922624500722303155285507305699689719921776:7940925676543875397236053787855908108301971735604816188258346045389094898659:6251324113868803887810489720303898206057027039878356780010250972997365420633', '18610384836050201850154911161375130158776466494967105643218942771825257615284:374983711236043917534518314230099351495389821421299379020233861590824184359:16631019648032875537843959305979611798948300093458140508344848956455484665057:1775533939933685303541316206912638084677760003793713587163717594332121785560', '17051413238459244747850569782102713872025163322505115503457201588386908477643:18374844977068704718276663561653076653562747738782855391253160037875164457396:9273861416516167362971906647498289391966692365940275070462836839397262110203:16112313008541511592743205022015360440037751502493708037157136506847795878774', '6834869487899280168398124820664870731309463865858570066128146591516094495760:7622445077437364834768651468839175193499568889033437340903850670710819413996:12032600475943243780610008306202525973855416537567075848204896171521405524382:7888555793313396733374582849253204459825385029175259893039791381540078648954', '15108567744595187631007881189706594871548400229494141464618456290475225281703:17499315251543787367456596454539662861812494156030861104859835937028120499107:13237965669983479563613270127263948669070988887401853227148939581708289233801:17947522842056779516152257986287295166108572608076133762541557937573752059527', '19259382161055078898663097663288417936824144611507279523322171166920908033289:14109503097051852876474699521907193802675114999675443259812001025711206550131:5158117661403121054086554735901161287570541058326318635217612866527395492200:3715322380648724761584242139609935920634736776088826855208563759886249878624', '16076877118896463297696938624294725456929492732064152589587819502431194445533:215230874172167297795540099472230340190822564063896001289370993061299898809:4849150298197063550858546079620409495661935308263096678759201142468032799768:13822974036605661261721590028039934405843242715456120643709618343487848482467'], 'status': 0}} [2023-09-07 18:14:19,340 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 23 -> 24 [2023-09-07 18:14:20,049 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:118 - Changing DKG status for free-basic-rosy-SKALE to DONE [2023-09-07 18:14:20,056 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.action:229 - Updating external state config [2023-09-07 18:14:20,057 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.action:190 - Creating new upstream_config rotation_id: 3, stream: regression-2.2 [2023-09-07 18:14:20,059 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.config.main:49 - Generating sChain config for free-basic-rosy-SKALE [2023-09-07 18:14:23,396 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110452, schain_finish_ts: 1692637668 [2023-09-07 18:14:23,913 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 9 -> 64 [2023-09-07 18:14:24,513 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:14:24,513 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:14:24,514 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:14:24,514 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:14:24,514 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:14:24,514 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:14:24,516 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:14:24,516 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 18:14:24,540 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:14:24,768 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:14:24,903 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 18:14:24,995 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:14:25,087 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:14:25,087 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:14:25,088 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:14:25,088 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 18:14:25,137 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:14:25,138 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2578 [2023-09-07 18:14:25,138 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:14:25,139 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:14:25,139 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:14:25,139 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:14:25,140 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:14:25,151 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:14:25,153 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:14:25.153783 [2023-09-07 18:14:25,262 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 18:14:25,262 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 18:14:25,265 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 18:14:25,273 ERROR][11][wan-tasty-kitalpha][T_1] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:14:25,273 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:14:25,324 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:14:25,357 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:14:25,392 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:14:25,396 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:14:25,405 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:14:25,405 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694110465.355742, 'te': 1694110465.362263, 'initial_status': True} ====================================================================================================  [2023-09-07 18:14:25,410 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:14:25.410263 [2023-09-07 18:14:25,416 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:14:25,419 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:14:25,933 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110464, schain_finish_ts: 0 [2023-09-07 18:14:26,199 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110464, schain_finish_ts: 1694108304 [2023-09-07 18:14:26,236 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:14:26,236 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:14:26,238 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:14:26.238288 [2023-09-07 18:14:26,247 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:14:26,247 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:14:26,258 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:14:26,267 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694110466.2452826, 'te': 1694110466.266825, 'initial_status': True} ====================================================================================================  [2023-09-07 18:14:26,268 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:14:26.268977 [2023-09-07 18:14:26,276 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:14:26,276 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:14:27,073 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:14:27,073 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:14:27,073 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 18:14:27,321 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:14:27,321 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:14:27,322 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 18:14:27,958 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110464, schain_finish_ts: 1693742004 [2023-09-07 18:14:28,914 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 35 -> 16 [2023-09-07 18:14:29,083 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:14:29,083 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:14:29,084 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:14:29,086 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 18:14:29,086 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 18:14:29,086 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:14:29,087 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:14:29,088 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 18:14:29,117 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:14:29,358 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:14:29,520 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:14:29,636 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:14:29,637 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2575 [2023-09-07 18:14:29,638 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:14:29,638 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:14:29,638 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:14:29,638 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:14:29,639 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:14:29,644 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:14:29,646 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:14:29.646819 [2023-09-07 18:14:29,881 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:14:29,913 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:14:29,947 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:14:29,949 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:14:29,957 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:14:29,958 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694110469.9114265, 'te': 1694110469.9187267, 'initial_status': True} ====================================================================================================  [2023-09-07 18:14:29,960 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:14:29.960304 [2023-09-07 18:14:29,965 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:14:29,966 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:14:30,877 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 18:14:30,877 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:14:30,879 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:14:30.879828 [2023-09-07 18:14:30,886 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:14:34,177 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 55 -> 63 [2023-09-07 18:14:35,351 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110464, schain_finish_ts: 1693412520 [2023-09-07 18:14:36,452 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:14:36,452 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:14:36,452 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:14:36,453 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:14:36,453 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:14:36,453 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:14:36,455 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:14:36,457 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 18:14:36,481 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:14:36,697 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:14:36,855 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:14:36,963 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:14:36,964 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2580 [2023-09-07 18:14:36,965 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:14:36,965 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:14:36,965 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:14:36,966 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:14:36,966 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:14:36,972 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:14:36,973 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:14:36.973975 [2023-09-07 18:14:37,231 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:14:37,258 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:14:37,285 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:14:37,287 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:14:37,300 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:14:37,300 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694110477.2567205, 'te': 1694110477.263299, 'initial_status': True} ====================================================================================================  [2023-09-07 18:14:37,301 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:14:37.301903 [2023-09-07 18:14:37,307 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:14:37,307 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:14:38,310 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:14:38,311 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:14:38,313 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:14:38.313523 [2023-09-07 18:14:38,332 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:14:38,332 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:14:38,344 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:14:38,352 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694110478.3293025, 'te': 1694110478.3519542, 'initial_status': True} ====================================================================================================  [2023-09-07 18:14:38,354 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:14:38.354798 [2023-09-07 18:14:38,362 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:14:38,369 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:14:39,807 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 14 -> 11 [2023-09-07 18:14:45,818 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 56 -> 60 [2023-09-07 18:14:47,393 INFO][1][MainProcess][MainThread] - core.schains.cleaner:65 - Cleaner process started [2023-09-07 18:14:47,420 INFO][31017][cleaner][MainThread] - core.schains.cleaner:110 - Cleaner procedure started. [2023-09-07 18:14:48,249 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110476, schain_finish_ts: 1692377364 [2023-09-07 18:14:49,353 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:14:49,354 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:14:49,354 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:14:49,354 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:14:49,354 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:14:49,355 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:14:49,356 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:14:49,358 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 18:14:49,380 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:14:49,614 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 18:14:49,614 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 18:14:49,739 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:14:49,877 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:14:49,992 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:14:49,993 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2563 [2023-09-07 18:14:49,994 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:14:49,994 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:14:49,994 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:14:49,995 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:14:49,995 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:14:50,002 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:14:50,004 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:14:50.004963 [2023-09-07 18:14:50,163 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:14:50,190 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:14:50,224 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:14:50,226 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:14:50,236 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:14:50,236 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694110490.1890473, 'te': 1694110490.1961474, 'initial_status': True} ====================================================================================================  [2023-09-07 18:14:50,238 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:14:50.238558 [2023-09-07 18:14:50,245 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:14:50,245 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:14:51,110 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:14:51,111 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:14:51,112 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:14:51.112882 [2023-09-07 18:14:51,120 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:14:51,121 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:14:51,130 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:14:51,137 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694110491.1195335, 'te': 1694110491.1370928, 'initial_status': True} ====================================================================================================  [2023-09-07 18:14:51,138 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:14:51.138951 [2023-09-07 18:14:51,144 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:14:51,144 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:14:51,866 ERROR][24912][submissive-well-off-rotanev][T_0] - core.schains.task:26 - Task submissive-well-off-rotanev-config failed with max() arg is an empty sequence Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 63, in execute self.am.upstream_config() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 194, in upstream_config new_config = create_new_upstream_config( File "/usr/src/admin/core/schains/config/main.py", line 51, in create_new_upstream_config schain_config = generate_schain_config_with_skale( File "/usr/src/admin/core/schains/config/generator.py", line 213, in generate_schain_config_with_skale node_groups = get_previous_schain_groups(skale, schain_name) File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 56, in get_previous_schain_groups _add_previous_schain_rotations_state( File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 119, in _add_previous_schain_rotations_state new_node_id = max(previous_nodes.items(), key=lambda x: x[1]['finish_ts'])[0] ValueError: max() arg is an empty sequence [2023-09-07 18:14:51,867 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:14:52,883 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110476, schain_finish_ts: 1693742304 [2023-09-07 18:14:53,009 INFO][31017][cleaner][MainThread] - core.schains.cleaner:116 - sChains on contracts: ['portly-passionate-sirius', 'curly-red-alterf', 'howling-hoedus', 'green-giddy-denebola', 'wan-tasty-kitalpha', 'free-basic-rosy-SKALE', 'submissive-well-off-rotanev'] sChains on node: ['curly-red-alterf', 'free-basic-rosy-SKALE', 'green-giddy-denebola', 'howling-hoedus', 'portly-passionate-sirius', 'submissive-well-off-rotanev', 'wan-tasty-kitalpha'] [2023-09-07 18:14:53,009 INFO][31017][cleaner][MainThread] - core.schains.cleaner:132 - Cleanup procedure finished [2023-09-07 18:14:53,013 INFO][1][MainProcess][MainThread] - core.schains.cleaner:67 - Cleaner process is joined. [2023-09-07 18:14:53,014 INFO][1][MainProcess][MainThread] - core.schains.cleaner:68 - Terminating the process [2023-09-07 18:14:53,014 INFO][1][MainProcess][MainThread] - __main__:70 - Sleeping for 90s after run_cleaner [2023-09-07 18:14:54,016 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:14:54,016 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:14:54,016 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:14:54,017 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 18:14:54,017 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 18:14:54,017 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:14:54,017 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:14:54,018 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 18:14:54,046 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:14:54,284 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:14:54,481 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:14:54,618 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:14:54,619 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2585 [2023-09-07 18:14:54,620 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:14:54,620 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:14:54,621 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:14:54,621 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:14:54,621 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:14:54,628 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:14:54,630 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:14:54.630471 [2023-09-07 18:14:54,805 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:14:54,830 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:14:54,860 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:14:54,863 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:14:54,871 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:14:54,872 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694110494.8284304, 'te': 1694110494.8346326, 'initial_status': True} ====================================================================================================  [2023-09-07 18:14:54,873 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:14:54.873949 [2023-09-07 18:14:54,878 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:14:54,879 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:14:55,004 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110476, schain_finish_ts: 0 [2023-09-07 18:14:55,795 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 18:14:55,795 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:14:55,798 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:14:55.797971 [2023-09-07 18:14:55,804 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:14:56,122 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:14:56,122 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:14:56,123 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:40 - Task submissive-well-off-rotanev-config finished with None [2023-09-07 18:14:56,123 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:42 - Running task submissive-well-off-rotanev-config [2023-09-07 18:14:56,123 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 18:14:57,261 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110476, schain_finish_ts: 1694108304 [2023-09-07 18:14:57,813 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': False} [2023-09-07 18:14:57,814 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:14:57,816 INFO][24912][submissive-well-off-rotanev][T_0] - web.models.schain:152 - Changing monitor_last_seen for submissive-well-off-rotanev to 2023-09-07 18:14:57.816061 [2023-09-07 18:14:57,821 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.action:190 - Creating new upstream_config rotation_id: 8, stream: regression-2.2 [2023-09-07 18:14:57,823 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.config.main:49 - Generating sChain config for submissive-well-off-rotanev [2023-09-07 18:14:58,385 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:14:58,385 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:14:58,385 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 18:15:02,466 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110476, schain_finish_ts: 1692637668 [2023-09-07 18:15:03,584 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:15:03,585 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:15:03,585 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:15:03,585 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:15:03,585 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:15:03,586 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:15:03,587 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:15:03,588 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 18:15:03,610 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:15:03,960 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:04,113 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:04,233 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:15:04,234 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2579 [2023-09-07 18:15:04,235 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:15:04,235 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:15:04,235 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:15:04,236 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:15:04,236 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:15:04,242 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:15:04,243 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:15:04.243925 [2023-09-07 18:15:04,405 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:04,432 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:15:04,464 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:15:04,466 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:15:04,474 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:15:04,475 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694110504.4311388, 'te': 1694110504.4386837, 'initial_status': True} ====================================================================================================  [2023-09-07 18:15:04,482 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:15:04.482640 [2023-09-07 18:15:04,488 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:15:04,488 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:15:05,035 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110476, schain_finish_ts: 1693742004 [2023-09-07 18:15:05,393 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:15:05,393 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:15:05,395 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:15:05.395903 [2023-09-07 18:15:05,405 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:15:05,406 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:15:05,413 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:15:05,418 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694110505.4038153, 'te': 1694110505.4187434, 'initial_status': True} ====================================================================================================  [2023-09-07 18:15:05,420 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:15:05.420708 [2023-09-07 18:15:05,425 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:15:05,425 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:15:06,162 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:15:06,163 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:15:06,163 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:15:06,163 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:15:06,164 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:15:06,165 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 18:15:06,193 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:15:06,469 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:06,815 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:06,937 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:15:06,938 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2576 [2023-09-07 18:15:06,939 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:15:06,939 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:15:06,939 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:15:06,940 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:15:06,940 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:15:06,947 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:15:06,951 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:15:06.951192 [2023-09-07 18:15:07,175 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:07,226 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:15:07,265 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:15:07,269 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:15:07,281 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:15:07,282 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694110507.2156708, 'te': 1694110507.2336478, 'initial_status': True} ====================================================================================================  [2023-09-07 18:15:07,284 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:15:07.284151 [2023-09-07 18:15:07,289 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:15:07,289 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:15:09,196 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 18:15:09,374 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:15:09,374 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 18:15:09,374 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 18:15:09,375 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 18:15:09,557 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 18:15:09,557 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 18:15:09,559 INFO][8][curly-red-alterf][T_1] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 18:15:09,564 ERROR][8][curly-red-alterf][T_1] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:15:09,564 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:15:14,421 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110512, schain_finish_ts: 1693412520 [2023-09-07 18:15:14,539 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 18:15:14,539 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 18:15:15,533 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:15:15,534 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:15:15,534 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:15:15,534 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:15:15,535 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:15:15,536 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:15:15,537 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:15:15,540 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 18:15:15,572 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:15:15,834 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:15,987 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:16,090 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:15:16,091 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2581 [2023-09-07 18:15:16,092 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:15:16,092 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:15:16,093 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:15:16,095 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:15:16,096 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:15:16,111 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:15:16,113 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:15:16.113565 [2023-09-07 18:15:16,274 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:16,300 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:15:16,333 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:15:16,335 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:15:16,343 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:15:16,344 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694110516.2993126, 'te': 1694110516.3080454, 'initial_status': True} ====================================================================================================  [2023-09-07 18:15:16,345 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:15:16.345764 [2023-09-07 18:15:16,352 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:15:16,352 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:15:17,285 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:15:17,285 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:15:17,287 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:15:17.287582 [2023-09-07 18:15:17,296 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:15:17,297 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:15:17,307 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:15:17,314 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694110517.2947652, 'te': 1694110517.3141584, 'initial_status': True} ====================================================================================================  [2023-09-07 18:15:17,316 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:15:17.316361 [2023-09-07 18:15:17,323 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:15:17,323 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:15:22,083 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110512, schain_finish_ts: 0 [2023-09-07 18:15:23,225 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:15:23,226 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:15:23,226 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 18:15:24,316 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110512, schain_finish_ts: 1694108304 [2023-09-07 18:15:25,442 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:15:25,443 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:15:25,443 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 18:15:26,272 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110524, schain_finish_ts: 1692377364 [2023-09-07 18:15:27,100 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110524, schain_finish_ts: 1693742004 [2023-09-07 18:15:27,382 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:15:27,383 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:15:27,383 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:15:27,383 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:15:27,383 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:15:27,384 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:15:27,385 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:15:27,386 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 18:15:27,412 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:15:27,596 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:27,750 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:27,862 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:15:27,863 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2564 [2023-09-07 18:15:27,864 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:15:27,864 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:15:27,864 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:15:27,865 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:15:27,865 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:15:27,872 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:15:27,878 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:15:27.878186 [2023-09-07 18:15:28,044 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:28,083 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:15:28,115 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:15:28,117 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:15:28,127 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:15:28,128 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694110528.0769088, 'te': 1694110528.0899572, 'initial_status': True} ====================================================================================================  [2023-09-07 18:15:28,130 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:15:28.130495 [2023-09-07 18:15:28,137 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:15:28,137 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:15:28,236 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:15:28,236 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:15:28,237 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:15:28,237 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 18:15:28,237 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 18:15:28,237 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:15:28,239 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:15:28,242 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 18:15:28,265 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:15:28,485 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:28,679 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:28,810 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:15:28,817 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2577 [2023-09-07 18:15:28,823 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:15:28,823 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:15:28,823 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:15:28,824 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:15:28,824 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:15:28,830 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:15:28,832 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:15:28.832744 [2023-09-07 18:15:29,014 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110524, schain_finish_ts: 1693742304 [2023-09-07 18:15:29,034 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:29,070 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:15:29,113 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:15:29,116 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:15:29,124 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:15:29,124 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694110529.0687225, 'te': 1694110529.075544, 'initial_status': True} ====================================================================================================  [2023-09-07 18:15:29,126 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:15:29.126419 [2023-09-07 18:15:29,131 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:15:29,132 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:15:29,168 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:15:29,169 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:15:29,171 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:15:29.171881 [2023-09-07 18:15:29,181 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:15:29,182 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:15:29,190 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:15:29,195 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694110529.179151, 'te': 1694110529.195544, 'initial_status': True} ====================================================================================================  [2023-09-07 18:15:29,197 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:15:29.197409 [2023-09-07 18:15:29,204 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:15:29,204 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:15:30,033 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 18:15:30,033 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:15:30,036 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:15:30.036068 [2023-09-07 18:15:30,042 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:15:30,138 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:15:30,138 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:15:30,139 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:15:30,139 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:15:30,139 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:15:30,139 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 18:15:30,185 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:15:30,423 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:30,783 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:30,940 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:15:30,942 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2586 [2023-09-07 18:15:30,943 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:15:30,943 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:15:30,943 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:15:30,944 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:15:30,944 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:15:30,951 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:15:30,953 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:15:30.953955 [2023-09-07 18:15:31,120 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:31,144 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:15:31,171 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:15:31,173 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:15:31,180 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:15:31,180 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694110531.1430275, 'te': 1694110531.1494603, 'initial_status': True} ====================================================================================================  [2023-09-07 18:15:31,182 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:15:31.182141 [2023-09-07 18:15:31,187 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:15:31,187 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:15:33,985 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 18:15:34,152 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:15:34,153 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:15:34,154 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:15:34,156 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 18:15:34,339 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 18:15:34,340 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 18:15:34,346 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 18:15:34,353 ERROR][11][wan-tasty-kitalpha][T_0] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:15:34,354 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:15:36,478 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110524, schain_finish_ts: 1693412520 [2023-09-07 18:15:37,588 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:15:37,588 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:15:37,588 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:15:37,588 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:15:37,589 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:15:37,589 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:15:37,590 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:15:37,591 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 18:15:37,621 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:15:37,808 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:38,060 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:38,164 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:15:38,166 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2582 [2023-09-07 18:15:38,166 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:15:38,166 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:15:38,167 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:15:38,167 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:15:38,167 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:15:38,174 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:15:38,178 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:15:38.178751 [2023-09-07 18:15:38,390 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:38,437 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:15:38,470 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:15:38,471 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:15:38,492 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:15:38,492 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694110538.436059, 'te': 1694110538.4458108, 'initial_status': True} ====================================================================================================  [2023-09-07 18:15:38,496 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:15:38.496052 [2023-09-07 18:15:38,496 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110524, schain_finish_ts: 1692637668 [2023-09-07 18:15:38,503 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:15:38,504 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:15:39,397 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:15:39,397 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:15:39,400 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:15:39.400641 [2023-09-07 18:15:39,411 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:15:39,411 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:15:39,419 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:15:39,425 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694110539.4089715, 'te': 1694110539.4248154, 'initial_status': True} ====================================================================================================  [2023-09-07 18:15:39,427 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:15:39.427620 [2023-09-07 18:15:39,433 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:15:39,433 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:15:39,592 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:15:39,592 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:15:39,592 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:15:39,592 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:15:39,593 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:15:39,593 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:15:39,594 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:15:39,595 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 40 [2023-09-07 18:15:39,618 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:15:39,807 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:39,983 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:40,105 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:15:40,106 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2580 [2023-09-07 18:15:40,107 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:15:40,107 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:15:40,108 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:15:40,108 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:15:40,109 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:15:40,116 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:15:40,118 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:15:40.118864 [2023-09-07 18:15:40,461 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:40,491 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:15:40,520 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:15:40,522 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:15:40,531 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:15:40,534 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694110540.4891038, 'te': 1694110540.4962516, 'initial_status': True} ====================================================================================================  [2023-09-07 18:15:40,537 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:15:40.537784 [2023-09-07 18:15:40,543 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:15:40,543 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:15:41,457 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:15:41,458 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:15:41,461 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:15:41.461764 [2023-09-07 18:15:41,469 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:15:41,470 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:15:41,478 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:15:41,484 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694110541.468196, 'te': 1694110541.4841316, 'initial_status': True} ====================================================================================================  [2023-09-07 18:15:41,485 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:15:41.485955 [2023-09-07 18:15:41,490 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:15:41,491 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:15:49,076 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 18:15:49,076 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 18:15:49,359 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110548, schain_finish_ts: 1694108304 [2023-09-07 18:15:50,301 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110548, schain_finish_ts: 1692377364 [2023-09-07 18:15:50,474 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:15:50,474 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:15:50,474 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 18:15:51,415 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:15:51,415 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:15:51,415 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:15:51,415 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:15:51,415 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:15:51,416 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:15:51,417 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:15:51,419 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 18:15:51,441 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:15:51,784 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:51,935 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:52,051 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:15:52,052 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2565 [2023-09-07 18:15:52,053 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:15:52,053 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:15:52,053 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:15:52,054 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:15:52,054 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:15:52,061 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:15:52,064 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:15:52.064516 [2023-09-07 18:15:52,177 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110548, schain_finish_ts: 1693742004 [2023-09-07 18:15:52,274 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:52,304 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:15:52,340 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:15:52,342 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:15:52,351 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:15:52,352 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694110552.3023334, 'te': 1694110552.3109202, 'initial_status': True} ====================================================================================================  [2023-09-07 18:15:52,353 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:15:52.353856 [2023-09-07 18:15:52,359 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:15:52,360 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:15:53,197 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:15:53,197 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:15:53,199 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:15:53.199670 [2023-09-07 18:15:53,208 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:15:53,208 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:15:53,216 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:15:53,223 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694110553.2067518, 'te': 1694110553.2229388, 'initial_status': True} ====================================================================================================  [2023-09-07 18:15:53,224 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:15:53.224862 [2023-09-07 18:15:53,229 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:15:53,229 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:15:53,313 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:15:53,313 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:15:53,313 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:15:53,314 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:15:53,314 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:15:53,314 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 18:15:53,336 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:15:53,609 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:54,045 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:54,216 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:15:54,219 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2578 [2023-09-07 18:15:54,219 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:15:54,220 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:15:54,220 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:15:54,221 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:15:54,222 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:15:54,236 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:15:54,240 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:15:54.240054 [2023-09-07 18:15:54,453 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:15:54,480 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:15:54,510 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:15:54,513 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:15:54,521 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:15:54,521 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694110554.4781313, 'te': 1694110554.4865491, 'initial_status': True} ====================================================================================================  [2023-09-07 18:15:54,523 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:15:54.523145 [2023-09-07 18:15:54,527 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:15:54,528 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:15:59,521 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110548, schain_finish_ts: 1693412520 [2023-09-07 18:16:00,655 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:16:00,656 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:16:00,656 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:16:00,656 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:16:00,656 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:16:00,657 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:16:00,658 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:16:00,660 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 18:16:00,687 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:16:00,888 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:01,056 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:01,160 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:16:01,161 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2583 [2023-09-07 18:16:01,162 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:16:01,162 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:16:01,163 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:16:01,163 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:16:01,163 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:16:01,169 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:16:01,171 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:16:01.171463 [2023-09-07 18:16:01,184 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110548, schain_finish_ts: 0 [2023-09-07 18:16:01,329 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:01,354 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:16:01,392 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:16:01,394 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:16:01,401 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:16:01,402 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694110561.353052, 'te': 1694110561.3611994, 'initial_status': True} ====================================================================================================  [2023-09-07 18:16:01,403 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:16:01.403905 [2023-09-07 18:16:01,408 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:16:01,409 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:16:02,099 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110548, schain_finish_ts: 1693742304 [2023-09-07 18:16:02,302 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:16:02,303 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:16:02,304 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 18:16:02,419 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:16:02,419 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:16:02,421 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:16:02.421288 [2023-09-07 18:16:02,428 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:16:02,428 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:16:02,435 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:16:02,441 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694110562.4273918, 'te': 1694110562.4409056, 'initial_status': True} ====================================================================================================  [2023-09-07 18:16:02,443 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:16:02.443577 [2023-09-07 18:16:02,448 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:16:02,448 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:16:03,262 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:16:03,262 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:16:03,262 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:16:03,262 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 18:16:03,262 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 18:16:03,263 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:16:03,264 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:16:03,264 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 18:16:03,291 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:16:03,534 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:03,770 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:03,890 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:16:03,891 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2587 [2023-09-07 18:16:03,892 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:16:03,892 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:16:03,892 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:16:03,893 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:16:03,893 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:16:03,900 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:16:03,902 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:16:03.902269 [2023-09-07 18:16:04,113 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:04,145 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:16:04,186 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:16:04,188 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:16:04,196 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:16:04,196 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694110564.1437736, 'te': 1694110564.1523175, 'initial_status': True} ====================================================================================================  [2023-09-07 18:16:04,198 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:16:04.198423 [2023-09-07 18:16:04,203 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:16:04,203 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:16:05,059 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 18:16:05,060 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:16:05,062 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:16:05.062210 [2023-09-07 18:16:05,066 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:16:08,506 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 18:16:08,687 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:16:08,687 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 18:16:08,688 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 18:16:08,688 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 18:16:08,857 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 18:16:08,858 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 18:16:08,860 INFO][8][curly-red-alterf][T_0] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 18:16:08,865 ERROR][8][curly-red-alterf][T_0] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:16:08,865 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:16:14,158 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.schain_config.rotation_history:40 - Collecting rotation history for free-basic-rosy-SKALE... [2023-09-07 18:16:15,277 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.schain_config.rotation_history:50 - Rotation data for free-basic-rosy-SKALE: Rotation(leaving_node_id=65, new_node_id=28, freeze_until=1694108100, rotation_counter=3) [2023-09-07 18:16:19,329 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110572, schain_finish_ts: 1692377364 [2023-09-07 18:16:20,430 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:16:20,430 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:16:20,430 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:16:20,430 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:16:20,431 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:16:20,431 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:16:20,433 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:16:20,433 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 18:16:20,464 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:16:20,497 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110572, schain_finish_ts: 1692637668 [2023-09-07 18:16:20,754 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:21,013 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:21,123 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:16:21,124 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2566 [2023-09-07 18:16:21,125 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:16:21,125 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:16:21,125 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:16:21,126 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:16:21,126 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:16:21,133 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:16:21,136 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:16:21.136127 [2023-09-07 18:16:21,236 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110572, schain_finish_ts: 1693742004 [2023-09-07 18:16:21,398 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:21,447 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:16:21,486 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:16:21,489 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:16:21,498 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:16:21,499 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694110581.4435704, 'te': 1694110581.4579701, 'initial_status': True} ====================================================================================================  [2023-09-07 18:16:21,501 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:16:21.501705 [2023-09-07 18:16:21,509 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:16:21,510 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:16:21,603 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:16:21,603 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:16:21,603 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:16:21,604 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:16:21,604 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:16:21,604 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:16:21,607 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:16:21,609 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 18:16:21,635 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:16:21,924 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:22,106 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:22,226 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:16:22,227 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2581 [2023-09-07 18:16:22,228 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:16:22,228 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:16:22,228 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:16:22,229 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:16:22,229 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:16:22,231 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:16:22,232 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:16:22,235 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:16:22.235406 [2023-09-07 18:16:22,241 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:16:22,244 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:16:22.244941 [2023-09-07 18:16:22,245 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:16:22,245 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:16:22,258 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:16:22,265 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694110582.2432632, 'te': 1694110582.2654974, 'initial_status': True} ====================================================================================================  [2023-09-07 18:16:22,269 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:16:22.269255 [2023-09-07 18:16:22,277 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:16:22,278 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:16:22,368 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:16:22,369 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:16:22,369 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:16:22,369 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 18:16:22,369 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 18:16:22,370 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:16:22,373 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:16:22,378 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 18:16:22,421 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:16:22,449 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:22,484 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:16:22,546 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:16:22,549 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:16:22,562 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:16:22,563 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694110582.4815094, 'te': 1694110582.4957938, 'initial_status': True} ====================================================================================================  [2023-09-07 18:16:22,565 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:16:22.565197 [2023-09-07 18:16:22,571 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:16:22,572 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:16:22,694 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:22,897 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:23,034 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:16:23,038 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2579 [2023-09-07 18:16:23,043 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:16:23,043 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:16:23,044 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:16:23,044 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:16:23,044 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:16:23,053 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:16:23,056 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:16:23.056194 [2023-09-07 18:16:23,067 INFO][1][MainProcess][MainThread] - core.schains.process_manager:57 - Process manager started [2023-09-07 18:16:23,202 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:23,227 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:16:23,255 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:16:23,256 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:16:23,265 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:16:23,266 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694110583.2258544, 'te': 1694110583.2325864, 'initial_status': True} ====================================================================================================  [2023-09-07 18:16:23,268 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:16:23.268033 [2023-09-07 18:16:23,272 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:16:23,273 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:16:23,277 INFO][1][MainProcess][MainThread] - core.schains.notifications:35 - Node account has 407295140193863165 WEI [2023-09-07 18:16:23,278 INFO][1][MainProcess][MainThread] - tools.notifications.messages:188 - Saving new balance state 3022 1 [2023-09-07 18:16:23,278 INFO][1][MainProcess][MainThread] - core.schains.process_manager:95 - Fetching schains to monitor... [2023-09-07 18:16:23,400 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:16:23,400 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:16:23,402 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:16:23.402479 [2023-09-07 18:16:23,410 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:16:23,410 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:16:23,418 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:16:23,426 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694110583.4083657, 'te': 1694110583.4267046, 'initial_status': True} ====================================================================================================  [2023-09-07 18:16:23,428 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:16:23.428879 [2023-09-07 18:16:23,434 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:16:23,434 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:16:23,524 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 18:16:23,524 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 18:16:24,100 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 18:16:24,100 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:16:24,102 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:16:24.102419 [2023-09-07 18:16:24,110 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:16:25,183 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110572, schain_finish_ts: 1693742304 [2023-09-07 18:16:26,307 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:16:26,307 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:16:26,308 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:16:26,309 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:16:26,311 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:16:26,313 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 18:16:26,336 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:16:26,577 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:26,874 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:26,977 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:16:26,978 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2588 [2023-09-07 18:16:26,979 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:16:26,979 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:16:26,979 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:16:26,980 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:16:26,980 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:16:26,987 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:16:26,990 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:16:26.990904 [2023-09-07 18:16:27,175 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:27,200 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:16:27,226 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:16:27,228 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:16:27,243 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:16:27,243 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694110587.1985312, 'te': 1694110587.2052078, 'initial_status': True} ====================================================================================================  [2023-09-07 18:16:27,245 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:16:27.245312 [2023-09-07 18:16:27,250 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:16:27,251 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:16:27,413 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110572, schain_finish_ts: 1694108304 [2023-09-07 18:16:28,538 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:16:28,539 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:16:28,539 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 34 [2023-09-07 18:16:28,862 INFO][1][MainProcess][MainThread] - core.schains.process_manager:109 - Get leaving_history for node ... [2023-09-07 18:16:29,015 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 31 -> 28 [2023-09-07 18:16:30,825 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110572, schain_finish_ts: 1692377616 [2023-09-07 18:16:32,778 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110572, schain_finish_ts: 0 [2023-09-07 18:16:33,146 INFO][1][MainProcess][MainThread] - core.schains.process_manager:117 - Got leaving sChains for the node: [] [2023-09-07 18:16:33,147 INFO][1][MainProcess][MainThread] - core.schains.process_manager:101 -  ==================================================================================================== Monitoring sChains Node ID: 16 sChains on node: [{'name': 'portly-passionate-sirius', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666281384, 'startBlock': 7803380, 'deposit': 0, 'index': 1, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x0000017723ca2', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'curly-red-alterf', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1676058936, 'startBlock': 8469646, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000001fe6cdfa', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'howling-hoedus', 'mainnetOwner': '0x1026467C91fdA40aCc7e6935d1E52f67274FD669', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666181400, 'startBlock': 7796660, 'deposit': 0, 'index': 0, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0xe5a600f4c2608', 'multitransactionMode': False, 'thresholdEncryption': False, 'active': True}, {'name': 'green-giddy-denebola', 'mainnetOwner': '0x8e835d5d9E28ffD259E8Fa8988Cacfc333A582B4', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674565164, 'startBlock': 8368049, 'deposit': 0, 'index': 4, 'generation': 1, 'originator': '0xd2001DAb6898127Be2F167B548691C87251D13C3', 'chainId': '0x00000585eb4b1', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'wan-tasty-kitalpha', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674568392, 'startBlock': 8368271, 'deposit': 0, 'index': 5, 'generation': 1, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x00000152e7be9', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'free-basic-rosy-SKALE', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1693996920, 'startBlock': 9646443, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000002bb90439', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'submissive-well-off-rotanev', 'mainnetOwner': '0x487360568fE5Ee4AE5c75782D5F63e9935B0E512', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1692361104, 'startBlock': 9539053, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000003791fa10', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}] Number of sChains on node: 7 Empty sChain structs: 0 ====================================================================================================  [2023-09-07 18:16:33,524 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:16:33,524 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain portly-passionate-sirius - Process is running: PID = 7 [2023-09-07 18:16:33,612 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110572, schain_finish_ts: 1693412520 [2023-09-07 18:16:33,902 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:16:33,902 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain curly-red-alterf - Process is running: PID = 8 [2023-09-07 18:16:34,282 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:16:34,282 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain howling-hoedus - Process is running: PID = 9 [2023-09-07 18:16:34,647 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 63 -> 16 [2023-09-07 18:16:34,651 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:16:34,651 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain green-giddy-denebola - Process is running: PID = 10 [2023-09-07 18:16:34,748 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:16:34,749 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:16:34,749 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:16:34,749 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:16:34,749 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:16:34,750 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:16:34,751 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:16:34,752 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 21 [2023-09-07 18:16:34,779 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:16:35,028 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:16:35,028 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain wan-tasty-kitalpha - Process is running: PID = 11 [2023-09-07 18:16:35,114 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:35,293 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:35,408 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:16:35,409 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2584 [2023-09-07 18:16:35,409 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:16:35,410 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:16:35,410 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:16:35,410 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:16:35,411 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:16:35,411 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:16:35,412 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain free-basic-rosy-SKALE - Process is running: PID = 30622 [2023-09-07 18:16:35,424 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:16:35,427 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:16:35.427783 [2023-09-07 18:16:35,608 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:35,641 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:16:35,672 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:16:35,675 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:16:35,688 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:16:35,694 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694110595.6394784, 'te': 1694110595.646506, 'initial_status': True} ====================================================================================================  [2023-09-07 18:16:35,699 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:16:35.699793 [2023-09-07 18:16:35,706 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:16:35,707 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:16:35,782 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:16:35,782 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain submissive-well-off-rotanev - Process is running: PID = 24912 [2023-09-07 18:16:35,783 INFO][1][MainProcess][MainThread] - core.schains.process_manager:88 - Process manager procedure finished [2023-09-07 18:16:35,783 INFO][1][MainProcess][MainThread] - __main__:65 - Sleeping for 90s after run_process_manager [2023-09-07 18:16:36,483 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:16:36,483 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:16:36,486 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:16:36.486535 [2023-09-07 18:16:36,496 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:16:36,496 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:16:36,502 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:16:36,507 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694110596.4949555, 'te': 1694110596.5071485, 'initial_status': True} ====================================================================================================  [2023-09-07 18:16:36,509 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:16:36.509073 [2023-09-07 18:16:36,514 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:16:36,514 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:16:40,650 INFO][30622][free-basic-rosy-SKALE][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 65 -> 57 [2023-09-07 18:16:40,865 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.config.generator:133 - Going to generate sChain config for free-basic-rosy-SKALE, node_name: regression-node16, node_id: 16, rotation_id: 3, ecdsa keyname: [SGX_KEY] [2023-09-07 18:16:41,192 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.action:209 - Saving new upstream config rotation_id: 3 [2023-09-07 18:16:41,202 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.config.main:65 - Going to change config_version for free-basic-rosy-SKALE: 0.0.0 -> regression-2.2 [2023-09-07 18:16:41,202 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:162 - Changing config_version for free-basic-rosy-SKALE to regression-2.2 [2023-09-07 18:16:41,213 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.config.main:65 - Going to change config_version for free-basic-rosy-SKALE: regression-2.2 -> regression-2.2 [2023-09-07 18:16:41,213 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:162 - Changing config_version for free-basic-rosy-SKALE to regression-2.2 [2023-09-07 18:16:41,220 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:190 - Changing sync_config_run for free-basic-rosy-SKALE to False [2023-09-07 18:16:41,227 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - free-basic-rosy-SKALE dkg: {'ts': 1694109752.3459191, 'te': 1694110460.0561116, 'initial_status': False} external_state: {'ts': 1694110460.0563388, 'te': 1694110460.0577118, 'initial_status': True} upstream_config: {'ts': 1694110460.0579245, 'te': 1694110601.2114797, 'initial_status': True} reset_config_record: {'ts': 1694110601.2114882, 'te': 1694110601.2267506, 'initial_status': True} ====================================================================================================  [2023-09-07 18:16:41,228 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:152 - Changing monitor_last_seen for free-basic-rosy-SKALE to 2023-09-07 18:16:41.228647 [2023-09-07 18:16:41,235 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:16:41,236 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:16:42,354 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110596, schain_finish_ts: 0 [2023-09-07 18:16:42,614 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 18:16:42,614 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 18:16:42,774 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 18:16:42,957 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:16:42,957 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:16:42,957 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:16:42,957 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 18:16:43,136 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 18:16:43,136 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 18:16:43,139 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 18:16:43,143 ERROR][11][wan-tasty-kitalpha][T_1] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:16:43,144 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:16:43,475 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:16:43,475 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:16:43,475 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 18:16:43,535 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110596, schain_finish_ts: 1692637668 [2023-09-07 18:16:44,318 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110596, schain_finish_ts: 1693742004 [2023-09-07 18:16:44,660 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:16:44,660 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:16:44,660 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:16:44,660 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:16:44,661 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:16:44,661 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:16:44,665 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:16:44,666 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 18:16:44,689 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:16:44,917 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:45,130 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:45,237 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:16:45,238 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2582 [2023-09-07 18:16:45,239 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:16:45,240 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:16:45,240 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:16:45,240 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:16:45,241 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:16:45,253 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:16:45,256 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:16:45.256275 [2023-09-07 18:16:45,461 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:16:45,462 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:16:45,462 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:16:45,463 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:16:45,463 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:16:45,464 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 18:16:45,487 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:16:45,543 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:45,573 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:16:45,606 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:16:45,609 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:16:45,617 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:16:45,618 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694110605.5712163, 'te': 1694110605.579497, 'initial_status': True} ====================================================================================================  [2023-09-07 18:16:45,620 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:16:45.620076 [2023-09-07 18:16:45,625 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:16:45,625 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:16:45,862 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:46,074 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:46,187 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:16:46,188 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2580 [2023-09-07 18:16:46,189 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:16:46,189 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:16:46,189 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:16:46,190 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:16:46,190 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:16:46,197 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:16:46,199 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:16:46.199155 [2023-09-07 18:16:46,387 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:46,419 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:16:46,455 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:16:46,458 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:16:46,468 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:16:46,469 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694110606.4165988, 'te': 1694110606.426365, 'initial_status': True} ====================================================================================================  [2023-09-07 18:16:46,471 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:16:46.471457 [2023-09-07 18:16:46,473 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:16:46,480 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:16:46,481 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:16:46,477 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:16:46,484 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:16:46.484332 [2023-09-07 18:16:46,493 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:16:46,494 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:16:46,502 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:16:46,509 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694110606.4916587, 'te': 1694110606.5091887, 'initial_status': True} ====================================================================================================  [2023-09-07 18:16:46,512 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:16:46.512057 [2023-09-07 18:16:46,518 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:16:46,518 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:16:47,256 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110596, schain_finish_ts: 1693742304 [2023-09-07 18:16:47,874 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:40 - Collecting rotation history for submissive-well-off-rotanev... [2023-09-07 18:16:48,377 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:16:48,377 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:16:48,377 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:16:48,377 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 18:16:48,378 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 18:16:48,378 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:16:48,379 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:16:48,381 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 18:16:48,403 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:16:48,593 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:48,739 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:48,838 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:16:48,839 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2589 [2023-09-07 18:16:48,839 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:16:48,840 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:16:48,840 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:16:48,840 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:16:48,840 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:16:48,846 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:16:48,848 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:16:48.848552 [2023-09-07 18:16:49,008 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:50 - Rotation data for submissive-well-off-rotanev: Rotation(leaving_node_id=65, new_node_id=65, freeze_until=1694108100, rotation_counter=8) [2023-09-07 18:16:49,011 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:49,037 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:16:49,076 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:16:49,078 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:16:49,089 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:16:49,090 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694110609.0358603, 'te': 1694110609.0427418, 'initial_status': True} ====================================================================================================  [2023-09-07 18:16:49,093 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:16:49.093378 [2023-09-07 18:16:49,098 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:16:49,100 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:16:50,119 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 18:16:50,119 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:16:50,121 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:16:50.121584 [2023-09-07 18:16:50,126 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:16:54,372 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110608, schain_finish_ts: 1692377364 [2023-09-07 18:16:55,493 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:16:55,493 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:16:55,494 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:16:55,494 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:16:55,494 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:16:55,494 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:16:55,496 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:16:55,498 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 18:16:55,519 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:16:55,714 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:55,868 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:55,987 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:16:55,988 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2567 [2023-09-07 18:16:55,990 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:16:55,990 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:16:55,991 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:16:55,991 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:16:55,992 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:16:56,004 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:16:56,011 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:16:56.011174 [2023-09-07 18:16:56,252 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:56,290 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:16:56,326 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:16:56,329 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:16:56,338 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:16:56,339 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694110616.287593, 'te': 1694110616.2964468, 'initial_status': True} ====================================================================================================  [2023-09-07 18:16:56,341 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:16:56.341462 [2023-09-07 18:16:56,348 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:16:56,350 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:16:56,696 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110608, schain_finish_ts: 1693412520 [2023-09-07 18:16:57,276 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:16:57,276 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:16:57,278 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:16:57.278327 [2023-09-07 18:16:57,286 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:16:57,287 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:16:57,295 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:16:57,300 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694110617.2848995, 'te': 1694110617.299858, 'initial_status': True} ====================================================================================================  [2023-09-07 18:16:57,302 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:16:57.302216 [2023-09-07 18:16:57,307 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:16:57,307 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:16:57,822 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:16:57,822 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:16:57,822 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:16:57,823 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:16:57,823 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:16:57,823 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:16:57,825 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:16:57,825 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 18:16:57,850 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:16:58,066 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:58,227 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:58,327 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:16:58,328 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2585 [2023-09-07 18:16:58,329 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:16:58,329 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:16:58,330 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:16:58,330 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:16:58,330 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:16:58,337 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:16:58,339 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:16:58.339468 [2023-09-07 18:16:58,512 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:16:58,540 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:16:58,579 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:16:58,582 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:16:58,592 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:16:58,593 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694110618.5376215, 'te': 1694110618.5479374, 'initial_status': True} ====================================================================================================  [2023-09-07 18:16:58,594 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:16:58.594723 [2023-09-07 18:16:58,600 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:16:58,601 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:16:59,586 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:16:59,586 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:16:59,588 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:16:59.588284 [2023-09-07 18:16:59,596 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:16:59,596 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:16:59,602 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:16:59,607 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694110619.5951858, 'te': 1694110619.6076298, 'initial_status': True} ====================================================================================================  [2023-09-07 18:16:59,609 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:16:59.609593 [2023-09-07 18:16:59,614 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:16:59,614 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:17:01,866 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 18:17:02,036 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:17:02,036 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 18:17:02,037 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 18:17:02,038 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 18:17:02,215 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 18:17:02,215 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 18:17:02,218 INFO][8][curly-red-alterf][T_1] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 18:17:02,223 ERROR][8][curly-red-alterf][T_1] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:17:02,223 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:17:03,474 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110620, schain_finish_ts: 1694108304 [2023-09-07 18:17:03,790 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 63 -> 65 [2023-09-07 18:17:04,586 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:17:04,587 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:17:04,587 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:17:04,587 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.task:40 - Task free-basic-rosy-SKALE-config finished with None [2023-09-07 18:17:04,587 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.task:42 - Running task free-basic-rosy-SKALE-config [2023-09-07 18:17:04,588 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.task:42 - Running task free-basic-rosy-SKALE-skaled [2023-09-07 18:17:04,592 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 18:17:04,615 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.firewall.utils:42 - Creating rule controller for free-basic-rosy-SKALE [2023-09-07 18:17:04,685 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': False, 'config': False, 'config_updated': False, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': False, 'ima_container': False, 'process': True, 'rotation_id_updated': False, 'rpc': False, 'skaled_container': False, 'upstream_exists': True, 'volume': False} [2023-09-07 18:17:04,686 INFO][30622][free-basic-rosy-SKALE][T_1] - tools.notifications.messages:129 - Sending checks notification with state [('blocks', False), ('config', False), ('config_updated', False), ('exit_code_ok', True), ('exit_zero', False), ('firewall_rules', False), ('ima_container', False), ('process', True), ('rotation_id_updated', False), ('rpc', False), ('skaled_container', False), ('upstream_exists', True), ('volume', False)] [2023-09-07 18:17:04,693 INFO][30622][free-basic-rosy-SKALE][T_1] - tools.notifications.messages:134 - Saving new checks state 1 [('blocks', False), ('config', False), ('config_updated', False), ('exit_code_ok', True), ('exit_zero', False), ('firewall_rules', False), ('ima_container', False), ('process', True), ('rotation_id_updated', False), ('rpc', False), ('skaled_container', False), ('upstream_exists', True), ('volume', False)] [2023-09-07 18:17:04,693 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/free-basic-rosy-SKALE/schain_free-basic-rosy-SKALE_3_1694110601.json [2023-09-07 18:17:04,694 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': False, 'config': False, 'config_updated': False, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': False, 'ima_container': False, 'process': True, 'rotation_id_updated': False, 'rpc': False, 'skaled_container': False, 'upstream_exists': True, 'volume': False} [2023-09-07 18:17:04,694 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:17:04,694 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type NoConfigSkaledMonitor starting [2023-09-07 18:17:04,697 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:152 - Changing monitor_last_seen for free-basic-rosy-SKALE to 2023-09-07 18:17:04.697481 [2023-09-07 18:17:04,703 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.skaled_monitor:182 - Creating skaled config [2023-09-07 18:17:04,703 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.action:413 - Syncing skaled config with upstream [2023-09-07 18:17:04,707 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:167 - Changing restart count for free-basic-rosy-SKALE to 0 [2023-09-07 18:17:04,714 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:172 - Changing failed rpc count for free-basic-rosy-SKALE to 0 [2023-09-07 18:17:04,720 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:127 - Changing first_run for free-basic-rosy-SKALE to False [2023-09-07 18:17:04,727 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:142 - Changing new_schain for free-basic-rosy-SKALE to False [2023-09-07 18:17:04,735 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:17:04,735 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - free-basic-rosy-SKALE update_config: {'ts': 1694110624.7033296, 'te': 1694110624.70485, 'initial_status': True} ====================================================================================================  [2023-09-07 18:17:04,737 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:152 - Changing monitor_last_seen for free-basic-rosy-SKALE to 2023-09-07 18:17:04.737193 [2023-09-07 18:17:04,742 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type NoConfigSkaledMonitor finished [2023-09-07 18:17:04,743 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:17:06,301 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:17:06,301 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:17:06,304 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:152 - Changing monitor_last_seen for free-basic-rosy-SKALE to 2023-09-07 18:17:06.304119 [2023-09-07 18:17:06,312 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.config.main:65 - Going to change config_version for free-basic-rosy-SKALE: regression-2.2 -> regression-2.2 [2023-09-07 18:17:06,312 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:162 - Changing config_version for free-basic-rosy-SKALE to regression-2.2 [2023-09-07 18:17:06,319 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:190 - Changing sync_config_run for free-basic-rosy-SKALE to False [2023-09-07 18:17:06,324 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - free-basic-rosy-SKALE reset_config_record: {'ts': 1694110626.3105292, 'te': 1694110626.3242953, 'initial_status': True} ====================================================================================================  [2023-09-07 18:17:06,326 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:152 - Changing monitor_last_seen for free-basic-rosy-SKALE to 2023-09-07 18:17:06.326355 [2023-09-07 18:17:06,331 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:17:06,331 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:17:08,588 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 23 -> 24 [2023-09-07 18:17:08,844 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 18:17:08,844 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 18:17:13,019 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 9 -> 64 [2023-09-07 18:17:13,636 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110632, schain_finish_ts: 1692637668 [2023-09-07 18:17:14,759 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:17:14,759 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:17:14,760 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:17:14,760 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:17:14,760 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:17:14,760 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:17:14,762 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:17:14,764 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 18:17:14,785 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:17:15,049 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:15,248 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:15,359 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:17:15,360 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2583 [2023-09-07 18:17:15,361 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:17:15,361 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:17:15,361 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:17:15,362 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:17:15,362 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:17:15,372 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:17:15,375 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:17:15.375502 [2023-09-07 18:17:15,550 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:15,578 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:17:15,624 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:17:15,627 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:17:15,634 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:17:15,635 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694110635.577143, 'te': 1694110635.5845778, 'initial_status': True} ====================================================================================================  [2023-09-07 18:17:15,637 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:17:15.637155 [2023-09-07 18:17:15,641 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:17:15,642 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:17:16,538 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:17:16,538 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:17:16,540 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:17:16.540665 [2023-09-07 18:17:16,549 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:17:16,550 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:17:16,557 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:17:16,562 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694110636.548357, 'te': 1694110636.5620224, 'initial_status': True} ====================================================================================================  [2023-09-07 18:17:16,564 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:17:16.564508 [2023-09-07 18:17:16,570 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:17:16,570 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:17:17,409 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110632, schain_finish_ts: 0 [2023-09-07 18:17:17,924 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 35 -> 16 [2023-09-07 18:17:18,521 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:17:18,526 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:17:18,532 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 18:17:20,785 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110632, schain_finish_ts: 1693412520 [2023-09-07 18:17:21,921 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:17:21,921 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:17:21,922 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:17:21,922 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:17:21,922 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:17:21,923 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:17:21,924 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:17:21,924 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 23 [2023-09-07 18:17:21,952 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:17:22,282 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:22,330 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110632, schain_finish_ts: 1693742304 [2023-09-07 18:17:22,422 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:22,529 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:17:22,530 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2586 [2023-09-07 18:17:22,531 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:17:22,531 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:17:22,531 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:17:22,532 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:17:22,532 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:17:22,538 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:17:22,540 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:17:22.540575 [2023-09-07 18:17:22,723 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:22,751 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:17:22,788 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:17:22,790 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:17:22,803 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:17:22,804 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694110642.749311, 'te': 1694110642.7583148, 'initial_status': True} ====================================================================================================  [2023-09-07 18:17:22,806 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:17:22.806055 [2023-09-07 18:17:22,814 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:17:22,818 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:17:23,287 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 55 -> 63 [2023-09-07 18:17:23,428 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110632, schain_finish_ts: 1693742004 [2023-09-07 18:17:23,471 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:17:23,472 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:17:23,472 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:17:23,472 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:17:23,473 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:17:23,473 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 18:17:23,500 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:17:23,745 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:17:23,745 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:17:23,747 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:17:23.747487 [2023-09-07 18:17:23,756 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:17:23,756 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:17:23,764 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:17:23,772 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694110643.7544458, 'te': 1694110643.7718284, 'initial_status': True} ====================================================================================================  [2023-09-07 18:17:23,774 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:17:23.774507 [2023-09-07 18:17:23,781 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:17:23,781 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:17:23,996 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:24,167 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:24,289 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:17:24,290 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2590 [2023-09-07 18:17:24,291 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:17:24,292 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:17:24,292 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:17:24,292 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:17:24,293 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:17:24,302 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:17:24,304 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:17:24.304288 [2023-09-07 18:17:24,452 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:24,478 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:17:24,507 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:17:24,509 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:17:24,517 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:17:24,517 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694110644.4765818, 'te': 1694110644.4826784, 'initial_status': True} ====================================================================================================  [2023-09-07 18:17:24,518 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:17:24.518959 [2023-09-07 18:17:24,523 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:17:24,524 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:17:24,574 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:17:24,575 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:17:24,575 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:17:24,575 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 18:17:24,575 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 18:17:24,576 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:17:24,577 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:17:24,582 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 18:17:24,611 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:17:24,845 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:25,009 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:25,140 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:17:25,148 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2581 [2023-09-07 18:17:25,149 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:17:25,149 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:17:25,149 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:17:25,150 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:17:25,150 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:17:25,158 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:17:25,162 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:17:25.162015 [2023-09-07 18:17:25,338 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:25,367 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:17:25,395 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:17:25,398 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:17:25,405 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:17:25,406 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694110645.3658783, 'te': 1694110645.3724818, 'initial_status': True} ====================================================================================================  [2023-09-07 18:17:25,408 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:17:25.408048 [2023-09-07 18:17:25,413 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:17:25,413 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:17:26,326 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 18:17:26,326 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:17:26,333 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:17:26.333304 [2023-09-07 18:17:26,345 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:17:28,223 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 18:17:28,423 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:17:28,424 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:17:28,426 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:17:28,426 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 18:17:28,604 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 18:17:28,605 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 18:17:28,609 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 18:17:28,621 ERROR][11][wan-tasty-kitalpha][T_0] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:17:28,621 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:17:29,025 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 14 -> 11 [2023-09-07 18:17:32,438 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110644, schain_finish_ts: 1692377364 [2023-09-07 18:17:33,553 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:17:33,553 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:17:33,554 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:17:33,554 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:17:33,554 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:17:33,554 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:17:33,556 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:17:33,556 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 18:17:33,586 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:17:33,794 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:33,974 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:34,098 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:17:34,100 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2568 [2023-09-07 18:17:34,100 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:17:34,101 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:17:34,101 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:17:34,102 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:17:34,103 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:17:34,111 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:17:34,114 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:17:34.114329 [2023-09-07 18:17:34,451 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:34,486 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:17:34,520 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:17:34,522 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:17:34,529 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:17:34,530 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694110654.484038, 'te': 1694110654.4921267, 'initial_status': True} ====================================================================================================  [2023-09-07 18:17:34,531 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:17:34.531511 [2023-09-07 18:17:34,535 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:17:34,536 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:17:35,158 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:142 - Adding rotation: 56 -> 60 [2023-09-07 18:17:35,385 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:17:35,385 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:17:35,387 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:17:35.387256 [2023-09-07 18:17:35,394 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:17:35,395 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:17:35,400 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:17:35,405 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694110655.3931336, 'te': 1694110655.4052675, 'initial_status': True} ====================================================================================================  [2023-09-07 18:17:35,407 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:17:35.407215 [2023-09-07 18:17:35,411 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:17:35,412 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:17:41,255 ERROR][24912][submissive-well-off-rotanev][T_0] - core.schains.task:26 - Task submissive-well-off-rotanev-config failed with max() arg is an empty sequence Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 63, in execute self.am.upstream_config() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 194, in upstream_config new_config = create_new_upstream_config( File "/usr/src/admin/core/schains/config/main.py", line 51, in create_new_upstream_config schain_config = generate_schain_config_with_skale( File "/usr/src/admin/core/schains/config/generator.py", line 213, in generate_schain_config_with_skale node_groups = get_previous_schain_groups(skale, schain_name) File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 56, in get_previous_schain_groups _add_previous_schain_rotations_state( File "/usr/local/lib/python3.9/site-packages/skale/schain_config/rotation_history.py", line 119, in _add_previous_schain_rotations_state new_node_id = max(previous_nodes.items(), key=lambda x: x[1]['finish_ts'])[0] ValueError: max() arg is an empty sequence [2023-09-07 18:17:41,255 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:17:41,526 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110656, schain_finish_ts: 1694108304 [2023-09-07 18:17:42,662 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:17:42,663 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:17:42,663 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:17:42,663 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.task:40 - Task free-basic-rosy-SKALE-config finished with None [2023-09-07 18:17:42,664 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.task:42 - Running task free-basic-rosy-SKALE-config [2023-09-07 18:17:42,664 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.task:40 - Task free-basic-rosy-SKALE-skaled finished with None [2023-09-07 18:17:42,665 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.task:42 - Running task free-basic-rosy-SKALE-skaled [2023-09-07 18:17:42,668 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 18:17:42,691 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.firewall.utils:42 - Creating rule controller for free-basic-rosy-SKALE [2023-09-07 18:17:42,696 ERROR][30622][free-basic-rosy-SKALE][T_0] - tools.helper:56 - Post request failed with: HTTPConnectionPool(host='127.0.0.1', port=10323): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) [2023-09-07 18:17:42,700 ERROR][30622][free-basic-rosy-SKALE][T_0] - tools.helper:56 - Post request failed with: HTTPConnectionPool(host='127.0.0.1', port=10323): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) [2023-09-07 18:17:43,228 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 82, redundant rules: 0 [2023-09-07 18:17:43,413 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 82, redundant rules: 0 [2023-09-07 18:17:43,463 ERROR][30622][free-basic-rosy-SKALE][T_0] - tools.helper:56 - Post request failed with: HTTPConnectionPool(host='127.0.0.1', port=10323): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) [2023-09-07 18:17:43,468 ERROR][30622][free-basic-rosy-SKALE][T_0] - tools.helper:56 - Post request failed with: HTTPConnectionPool(host='127.0.0.1', port=10323): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) [2023-09-07 18:17:43,521 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': False, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': False, 'ima_container': False, 'process': True, 'rotation_id_updated': True, 'rpc': False, 'skaled_container': False, 'upstream_exists': True, 'volume': False} [2023-09-07 18:17:43,523 INFO][30622][free-basic-rosy-SKALE][T_0] - tools.notifications.messages:129 - Sending checks notification with state [('blocks', False), ('config', True), ('config_updated', True), ('exit_code_ok', True), ('exit_zero', False), ('firewall_rules', False), ('ima_container', False), ('process', True), ('rotation_id_updated', True), ('rpc', False), ('skaled_container', False), ('upstream_exists', True), ('volume', False)] [2023-09-07 18:17:43,526 INFO][30622][free-basic-rosy-SKALE][T_0] - tools.notifications.messages:134 - Saving new checks state 1 [('blocks', False), ('config', True), ('config_updated', True), ('exit_code_ok', True), ('exit_zero', False), ('firewall_rules', False), ('ima_container', False), ('process', True), ('rotation_id_updated', True), ('rpc', False), ('skaled_container', False), ('upstream_exists', True), ('volume', False)] [2023-09-07 18:17:43,527 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/free-basic-rosy-SKALE/schain_free-basic-rosy-SKALE_3_1694110601.json [2023-09-07 18:17:43,527 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': False, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': False, 'ima_container': False, 'process': True, 'rotation_id_updated': True, 'rpc': False, 'skaled_container': False, 'upstream_exists': True, 'volume': False} [2023-09-07 18:17:43,528 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:17:43,534 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:17:43,536 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:152 - Changing monitor_last_seen for free-basic-rosy-SKALE to 2023-09-07 18:17:43.536774 [2023-09-07 18:17:43,685 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 82, redundant rules: 0 [2023-09-07 18:17:43,874 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 82, redundant rules: 0 [2023-09-07 18:17:43,875 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.action:276 - Configuring firewall rules [2023-09-07 18:17:43,878 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.action:286 - Adding ranges [IpRange(start_ip='54.70.182.107', end_ip='54.70.182.107')] [2023-09-07 18:17:43,879 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.firewall.rule_controller:201 - Syncing firewall rules [2023-09-07 18:17:44,685 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:17:44,694 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:17:44,707 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:152 - Changing monitor_last_seen for free-basic-rosy-SKALE to 2023-09-07 18:17:44.707723 [2023-09-07 18:17:44,726 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.config.main:65 - Going to change config_version for free-basic-rosy-SKALE: regression-2.2 -> regression-2.2 [2023-09-07 18:17:44,726 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:162 - Changing config_version for free-basic-rosy-SKALE to regression-2.2 [2023-09-07 18:17:44,738 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:190 - Changing sync_config_run for free-basic-rosy-SKALE to False [2023-09-07 18:17:44,748 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - free-basic-rosy-SKALE reset_config_record: {'ts': 1694110664.7243407, 'te': 1694110664.747973, 'initial_status': True} ====================================================================================================  [2023-09-07 18:17:44,763 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:152 - Changing monitor_last_seen for free-basic-rosy-SKALE to 2023-09-07 18:17:44.763519 [2023-09-07 18:17:44,773 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:17:44,777 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:17:45,280 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 18:17:45,280 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 18:17:45,841 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110656, schain_finish_ts: 1693412520 [2023-09-07 18:17:46,960 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:17:46,960 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:17:46,960 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:17:46,961 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:17:46,961 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:17:46,961 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:17:46,963 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:17:46,963 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 18:17:46,989 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:17:47,402 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:47,649 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:47,774 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:17:47,775 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2587 [2023-09-07 18:17:47,776 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:17:47,776 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:17:47,776 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:17:47,777 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:17:47,777 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:17:47,783 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:17:47,786 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:17:47.785992 [2023-09-07 18:17:48,129 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:48,159 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:17:48,199 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:17:48,201 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:17:48,208 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:17:48,209 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694110668.157227, 'te': 1694110668.1675556, 'initial_status': True} ====================================================================================================  [2023-09-07 18:17:48,210 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:17:48.210648 [2023-09-07 18:17:48,216 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:17:48,216 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:17:48,732 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110656, schain_finish_ts: 1692637668 [2023-09-07 18:17:48,811 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:17:48,811 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:17:48,813 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:17:48.813895 [2023-09-07 18:17:48,821 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:17:48,821 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:17:48,828 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:17:48,833 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694110668.8195791, 'te': 1694110668.8334243, 'initial_status': True} ====================================================================================================  [2023-09-07 18:17:48,835 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:17:48.835632 [2023-09-07 18:17:48,843 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:17:48,843 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:17:49,876 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:17:49,876 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:17:49,876 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:17:49,876 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:17:49,877 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:17:49,877 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:17:49,878 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:17:49,880 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 31 [2023-09-07 18:17:49,910 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:17:50,425 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:50,830 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:50,930 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:17:50,931 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2584 [2023-09-07 18:17:50,932 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:17:50,933 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:17:50,933 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:17:50,933 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:17:50,934 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:17:50,940 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:17:50,942 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:17:50.942258 [2023-09-07 18:17:51,196 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:51,227 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:17:51,269 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:17:51,271 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:17:51,281 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:17:51,284 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694110671.2258484, 'te': 1694110671.2339675, 'initial_status': True} ====================================================================================================  [2023-09-07 18:17:51,292 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:17:51.292636 [2023-09-07 18:17:51,300 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:17:51,301 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:17:51,460 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110668, schain_finish_ts: 0 [2023-09-07 18:17:51,731 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:17:51,731 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:17:51,733 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:17:51.733873 [2023-09-07 18:17:51,742 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:17:51,742 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:17:51,748 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:17:51,753 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694110671.7399938, 'te': 1694110671.752915, 'initial_status': True} ====================================================================================================  [2023-09-07 18:17:51,754 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:17:51.754890 [2023-09-07 18:17:51,760 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:17:51,761 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:17:52,570 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:17:52,571 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:17:52,571 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:40 - Task submissive-well-off-rotanev-config finished with None [2023-09-07 18:17:52,571 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.task:42 - Running task submissive-well-off-rotanev-config [2023-09-07 18:17:52,572 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 18:17:53,419 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110668, schain_finish_ts: 1693742304 [2023-09-07 18:17:53,524 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110668, schain_finish_ts: 1693742004 [2023-09-07 18:17:54,311 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': False} [2023-09-07 18:17:54,312 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:17:54,317 INFO][24912][submissive-well-off-rotanev][T_0] - web.models.schain:152 - Changing monitor_last_seen for submissive-well-off-rotanev to 2023-09-07 18:17:54.317762 [2023-09-07 18:17:54,322 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.monitor.action:190 - Creating new upstream_config rotation_id: 8, stream: regression-2.2 [2023-09-07 18:17:54,324 INFO][24912][submissive-well-off-rotanev][T_0] - core.schains.config.main:49 - Generating sChain config for submissive-well-off-rotanev [2023-09-07 18:17:54,569 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:17:54,569 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:17:54,570 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:17:54,570 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 18:17:54,570 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 18:17:54,570 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:17:54,572 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:17:54,574 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 33 [2023-09-07 18:17:54,597 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:17:54,722 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:17:54,722 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:17:54,723 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:17:54,723 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:17:54,723 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:17:54,724 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 29 [2023-09-07 18:17:54,749 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:17:54,913 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:55,307 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:55,571 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:55,675 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:17:55,677 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2591 [2023-09-07 18:17:55,677 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:17:55,678 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:17:55,678 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:17:55,678 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:17:55,679 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:17:55,692 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:17:55,700 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:17:55.700286 [2023-09-07 18:17:55,833 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:55,958 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:17:55,959 INFO][8][curly-red-alterf][T_1] - tools.notifications.messages:142 - Saving new checks count 2582 [2023-09-07 18:17:55,960 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:17:55,960 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:17:55,960 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:17:55,961 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:17:55,962 INFO][8][curly-red-alterf][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:17:55,970 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:17:55,971 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:17:55.971866 [2023-09-07 18:17:56,172 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:56,204 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:17:56,214 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.action:266 - Creating volume [2023-09-07 18:17:56,236 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:17:56,238 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:17:56,242 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.volume:47 - Creating volume for schain: free-basic-rosy-SKALE [2023-09-07 18:17:56,243 INFO][30622][free-basic-rosy-SKALE][T_0] - root:122 - Creating volume - size: 22199992320, name: free-basic-rosy-SKALE, driver_opts: {'size': '22199992320'} [2023-09-07 18:17:56,259 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:17:56,259 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694110676.2023256, 'te': 1694110676.209054, 'initial_status': True} ====================================================================================================  [2023-09-07 18:17:56,261 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:17:56.261264 [2023-09-07 18:17:56,267 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:17:56,267 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:17:56,308 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 18:17:56,308 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:17:56,311 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:17:56.311215 [2023-09-07 18:17:56,316 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:17:56,377 INFO][8][curly-red-alterf][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:17:56,547 INFO][8][curly-red-alterf][T_1] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:17:56,559 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.action:303 - Starting skaled container watchman snapshot: False, start_ts: None [2023-09-07 18:17:56,561 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.containers:60 - Monitoring container for sChain free-basic-rosy-SKALE [2023-09-07 18:17:56,581 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.containers:74 - SChain free-basic-rosy-SKALE: container doesn't exits [2023-09-07 18:17:56,587 INFO][8][curly-red-alterf][T_1] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:17:56,590 INFO][8][curly-red-alterf][T_1] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:17:56,597 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:17:56,597 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694110676.545304, 'te': 1694110676.5557992, 'initial_status': True} ====================================================================================================  [2023-09-07 18:17:56,599 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:17:56.599135 [2023-09-07 18:17:56,603 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:17:56,604 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:17:56,635 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.runner:149 -  ==================================================================================================== Running container... Container name: skale_schain_free-basic-rosy-SKALE Image name: skalenetwork/schain:3.17.0-beta.8 Args: {'security_opt': ['seccomp=unconfined'], 'restart_policy': {'MaximumRetryCount': 0, 'Name': 'no'}, 'network': 'host', 'cap_add': ['SYS_PTRACE', 'SYS_ADMIN', 'SYS_NICE'], 'volumes': {'/root/.skale/node_data': {'bind': '/skale_node_data', 'mode': 'ro'}, '/root/.skale': {'bind': '/skale_vol', 'mode': 'ro'}, '/root/.skale/node_data/schains/free-basic-rosy-SKALE': {'bind': '/schain_config', 'mode': 'rw'}, 'free-basic-rosy-SKALE': {'bind': '/data_dir', 'mode': 'rw'}, 'shared-space': {'bind': '/shared-space', 'mode': 'rw'}}, 'log_config': {'Type': 'json-file', 'Config': {'max-size': '250m', 'max-file': '5'}}, 'ulimits': [{'Name': 'core', 'Soft': -1, 'Hard': -1}], 'cpu_shares': 102, 'mem_limit': 2664121630, 'environment': {'SEGFAULT_SIGNALS': 'all'}, 'command': '--config /schain_config/schain_free-basic-rosy-SKALE.json -d /data_dir --ipcpath /data_dir --http-port 10323 --https-port 10328 --ws-port 10322 --wss-port 10327 --sgx-url https://[SGX_IP]:1026 --shared-space-path /shared-space/data --main-net-url http://[ETH_IP]:8888/goerli -v 3 --web3-trace --enable-debug-behavior-apis --aa no --ssl-key /skale_node_data/ssl/ssl_key --ssl-cert /skale_node_data/ssl/ssl_cert'} ====================================================================================================  [2023-09-07 18:17:56,936 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.runner:152 -  ==================================================================================================== Container created Container name: skale_schain_free-basic-rosy-SKALE Container id: e239be470a5237c040937a9400faab452eb124015e2d6b7be4030eed19d437ba ====================================================================================================  [2023-09-07 18:17:56,936 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:182 - Resetting failed counters for free-basic-rosy-SKALE [2023-09-07 18:17:56,937 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:167 - Changing restart count for free-basic-rosy-SKALE to 0 [2023-09-07 18:17:56,941 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:172 - Changing failed rpc count for free-basic-rosy-SKALE to 0 [2023-09-07 18:18:04,854 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 18:18:05,033 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:18:05,034 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 18:18:05,034 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 18:18:05,034 INFO][8][curly-red-alterf][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 18:18:05,208 INFO][8][curly-red-alterf][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 18:18:05,209 INFO][8][curly-red-alterf][T_0] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 18:18:05,211 INFO][8][curly-red-alterf][T_0] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 18:18:05,216 ERROR][8][curly-red-alterf][T_0] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:18:05,216 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:18:05,799 INFO][1][MainProcess][MainThread] - core.schains.cleaner:65 - Cleaner process started [2023-09-07 18:18:05,825 INFO][31071][cleaner][MainThread] - core.schains.cleaner:110 - Cleaner procedure started. [2023-09-07 18:18:07,494 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110680, schain_finish_ts: 1692377364 [2023-09-07 18:18:08,609 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:18:08,609 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:18:08,610 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:18:08,611 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:18:08,611 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:18:08,612 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:18:08,613 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:18:08,615 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 18:18:08,648 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:18:08,890 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:09,124 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:09,251 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:18:09,252 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2569 [2023-09-07 18:18:09,252 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:18:09,253 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:18:09,253 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:18:09,253 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:18:09,254 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:18:09,260 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:18:09,262 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:18:09.262807 [2023-09-07 18:18:09,602 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:09,640 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:18:09,674 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:18:09,676 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:18:09,691 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:18:09,693 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694110689.638186, 'te': 1694110689.6448815, 'initial_status': True} ====================================================================================================  [2023-09-07 18:18:09,703 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:18:09.703640 [2023-09-07 18:18:09,708 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:18:09,709 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:18:10,382 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:18:10,382 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:18:10,385 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:18:10.385234 [2023-09-07 18:18:10,392 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:18:10,393 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:18:10,399 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:18:10,404 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694110690.3912697, 'te': 1694110690.4037635, 'initial_status': True} ====================================================================================================  [2023-09-07 18:18:10,405 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:18:10.405694 [2023-09-07 18:18:10,409 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:18:10,410 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:18:11,481 INFO][31071][cleaner][MainThread] - core.schains.cleaner:116 - sChains on contracts: ['portly-passionate-sirius', 'curly-red-alterf', 'howling-hoedus', 'green-giddy-denebola', 'wan-tasty-kitalpha', 'free-basic-rosy-SKALE', 'submissive-well-off-rotanev'] sChains on node: ['curly-red-alterf', 'free-basic-rosy-SKALE', 'green-giddy-denebola', 'howling-hoedus', 'portly-passionate-sirius', 'submissive-well-off-rotanev', 'wan-tasty-kitalpha'] [2023-09-07 18:18:11,482 INFO][31071][cleaner][MainThread] - core.schains.cleaner:132 - Cleanup procedure finished [2023-09-07 18:18:11,487 INFO][1][MainProcess][MainThread] - core.schains.cleaner:67 - Cleaner process is joined. [2023-09-07 18:18:11,487 INFO][1][MainProcess][MainThread] - core.schains.cleaner:68 - Terminating the process [2023-09-07 18:18:11,487 INFO][1][MainProcess][MainThread] - __main__:70 - Sleeping for 90s after run_cleaner [2023-09-07 18:18:11,615 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110680, schain_finish_ts: 1694108304 [2023-09-07 18:18:12,754 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:18:12,755 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:18:12,755 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:18:12,755 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.task:40 - Task free-basic-rosy-SKALE-config finished with None [2023-09-07 18:18:12,755 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.task:42 - Running task free-basic-rosy-SKALE-config [2023-09-07 18:18:12,756 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 18:18:14,482 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:18:14,482 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:18:14,484 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:152 - Changing monitor_last_seen for free-basic-rosy-SKALE to 2023-09-07 18:18:14.484453 [2023-09-07 18:18:14,494 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.config.main:65 - Going to change config_version for free-basic-rosy-SKALE: regression-2.2 -> regression-2.2 [2023-09-07 18:18:14,494 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:162 - Changing config_version for free-basic-rosy-SKALE to regression-2.2 [2023-09-07 18:18:14,500 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:190 - Changing sync_config_run for free-basic-rosy-SKALE to False [2023-09-07 18:18:14,505 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - free-basic-rosy-SKALE reset_config_record: {'ts': 1694110694.4926527, 'te': 1694110694.5053413, 'initial_status': True} ====================================================================================================  [2023-09-07 18:18:14,508 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:152 - Changing monitor_last_seen for free-basic-rosy-SKALE to 2023-09-07 18:18:14.508185 [2023-09-07 18:18:14,512 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:18:14,513 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:18:15,168 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 18:18:15,168 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 18:18:17,000 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.action:392 - Running IMA container watchman [2023-09-07 18:18:17,000 INFO][30622][free-basic-rosy-SKALE][T_0] - core.ima.schain:48 - Copying /skale_vol/contracts_info/schain_ima_abi.json -> /skale_node_data/schains/free-basic-rosy-SKALE/schain_ima_abi.json [2023-09-07 18:18:17,005 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.containers:145 - free-basic-rosy-SKALE No IMA container, creating, image skalenetwork/ima:2.0.0-beta.10 [2023-09-07 18:18:17,022 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.runner:149 -  ==================================================================================================== Running container... Container name: skale_ima_free-basic-rosy-SKALE Image name: skalenetwork/ima:2.0.0-beta.10 Args: {'restart_policy': {'Name': 'unless-stopped'}, 'network': 'host', 'volumes': {'/root/.skale/node_data': {'bind': '/skale_node_data', 'mode': 'ro'}, '/root/.skale': {'bind': '/skale_vol', 'mode': 'ro'}, '/root/.skale/node_data/schains/free-basic-rosy-SKALE': {'bind': '/schain_config', 'mode': 'rw'}}, 'log_config': {'Type': 'json-file', 'Config': {'max-size': '250m', 'max-file': '5'}}, 'cpu_shares': 25, 'mem_limit': 666030407, 'environment': {'SCHAIN_DIR': '/skale_node_data/schains/free-basic-rosy-SKALE', 'MANAGER_ABI_PATH': '/skale_vol/contracts_info/manager.json', 'MAINNET_PROXY_PATH': '/skale_vol/contracts_info/ima.json', 'SCHAIN_PROXY_PATH': '/skale_node_data/schains/free-basic-rosy-SKALE/schain_ima_abi.json', 'STATE_FILE': '/schain_config/ima_state.json', 'SCHAIN_NAME': 'free-basic-rosy-SKALE', 'SCHAIN_RPC_URL': 'http://127.0.0.1:10323', 'MAINNET_RPC_URL': 'http://[ETH_IP]:8888/goerli', 'NODE_NUMBER': 14, 'NODES_COUNT': 16, 'SGX_URL': 'https://[SGX_IP]:1026', 'ECDSA_KEY_NAME': '[SGX_KEY]', 'BLS_KEY_NAME': 'BLS_KEY:SCHAIN_ID:18860195493846376099765944407454631582519828060563062387235736098366005:NODE_ID:14:DKG_ID:3', 'SGX_SSL_KEY_PATH': '/skale_node_data/sgx_certs/sgx.key', 'SGX_SSL_CERT_PATH': '/skale_node_data/sgx_certs/sgx.crt', 'NODE_ADDRESS': '0x523e13d962a7614752efa3db62e0466df0d9e89d', 'TM_URL_MAIN_NET': 'redis://@127.0.0.1:6379', 'CID_MAIN_NET': 5, 'CID_SCHAIN': '0x000002bb90439', 'MONITORING_PORT': 10326, 'RPC_PORT': 10330, 'TIME_FRAMING': 1800}} ====================================================================================================  [2023-09-07 18:18:17,244 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.runner:152 -  ==================================================================================================== Container created Container name: skale_ima_free-basic-rosy-SKALE Container id: a54c226d4056e745b79b599b5d9bc40eab03058a113fa6204f24d0cb33b28e03 ====================================================================================================  [2023-09-07 18:18:17,249 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:127 - Changing first_run for free-basic-rosy-SKALE to False [2023-09-07 18:18:17,252 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:142 - Changing new_schain for free-basic-rosy-SKALE to False [2023-09-07 18:18:17,259 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:18:17,259 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - free-basic-rosy-SKALE firewall_rules: {'ts': 1694110663.685905, 'te': 1694110676.1752875, 'initial_status': False} volume: {'ts': 1694110676.1951754, 'te': 1694110676.557208, 'initial_status': False} skaled_container: {'ts': 1694110676.5592556, 'te': 1694110696.9621277, 'initial_status': True} ====================================================================================================  [2023-09-07 18:18:17,261 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:152 - Changing monitor_last_seen for free-basic-rosy-SKALE to 2023-09-07 18:18:17.261950 [2023-09-07 18:18:17,267 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:18:17,268 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:18:17,912 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110692, schain_finish_ts: 1693412520 [2023-09-07 18:18:19,032 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:18:19,032 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:18:19,033 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:18:19,033 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:18:19,033 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:18:19,033 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:18:19,035 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:18:19,036 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 18:18:19,068 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:18:19,338 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:19,510 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:19,626 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:18:19,627 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2588 [2023-09-07 18:18:19,633 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:18:19,634 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:18:19,634 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:18:19,635 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:18:19,635 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:18:19,642 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:18:19,645 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:18:19.645672 [2023-09-07 18:18:19,853 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:19,882 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:18:19,918 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:18:19,921 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:18:19,928 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:18:19,929 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694110699.880114, 'te': 1694110699.8889103, 'initial_status': True} ====================================================================================================  [2023-09-07 18:18:19,930 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:18:19.930711 [2023-09-07 18:18:19,939 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:18:19,940 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:18:20,813 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:18:20,814 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:18:20,816 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:18:20.815989 [2023-09-07 18:18:20,823 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:18:20,823 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:18:20,829 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:18:20,833 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694110700.8218062, 'te': 1694110700.8334248, 'initial_status': True} ====================================================================================================  [2023-09-07 18:18:20,835 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:18:20.835148 [2023-09-07 18:18:20,840 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:18:20,840 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:18:21,511 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110692, schain_finish_ts: 0 [2023-09-07 18:18:21,804 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110692, schain_finish_ts: 1692637668 [2023-09-07 18:18:22,697 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:18:22,698 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:18:22,699 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 28 [2023-09-07 18:18:22,983 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:18:22,983 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:18:22,983 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:18:22,984 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:18:22,984 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:18:22,984 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:18:22,986 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:18:22,987 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 18:18:23,017 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:18:23,242 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:23,426 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:23,544 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:18:23,545 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2585 [2023-09-07 18:18:23,546 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:18:23,547 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:18:23,547 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:18:23,548 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:18:23,548 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:18:23,559 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:18:23,567 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:18:23.567134 [2023-09-07 18:18:23,786 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:23,815 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:18:23,851 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:18:23,853 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:18:23,861 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:18:23,861 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694110703.8135335, 'te': 1694110703.820819, 'initial_status': True} ====================================================================================================  [2023-09-07 18:18:23,863 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:18:23.863474 [2023-09-07 18:18:23,868 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:18:23,868 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:18:24,688 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110692, schain_finish_ts: 1693742004 [2023-09-07 18:18:24,748 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:18:24,749 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:18:24,752 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:18:24.752229 [2023-09-07 18:18:24,760 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:18:24,760 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:18:24,768 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:18:24,773 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694110704.7587333, 'te': 1694110704.7728975, 'initial_status': True} ====================================================================================================  [2023-09-07 18:18:24,776 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:18:24.776217 [2023-09-07 18:18:24,780 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:18:24,781 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:18:25,845 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:18:25,845 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:18:25,846 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:18:25,846 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 18:18:25,847 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 18:18:25,847 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:18:25,849 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:18:25,854 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 18:18:25,885 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:18:26,102 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:26,288 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:26,408 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:18:26,409 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2583 [2023-09-07 18:18:26,410 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:18:26,411 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:18:26,411 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:18:26,411 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:18:26,412 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:18:26,418 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:18:26,420 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:18:26.420461 [2023-09-07 18:18:26,775 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:26,803 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:18:26,856 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:18:26,859 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:18:26,868 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:18:26,869 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694110706.8022122, 'te': 1694110706.8081954, 'initial_status': True} ====================================================================================================  [2023-09-07 18:18:26,871 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:18:26.871667 [2023-09-07 18:18:26,876 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:18:26,877 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:18:27,656 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 18:18:27,656 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:18:27,658 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:18:27.658042 [2023-09-07 18:18:27,663 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:18:28,523 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110704, schain_finish_ts: 1693742304 [2023-09-07 18:18:29,652 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:18:29,652 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:18:29,652 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:18:29,653 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:18:29,653 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:18:29,653 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 22 [2023-09-07 18:18:29,688 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:18:30,193 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:30,425 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:30,544 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:18:30,545 INFO][11][wan-tasty-kitalpha][T_0] - tools.notifications.messages:142 - Saving new checks count 2592 [2023-09-07 18:18:30,545 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:18:30,545 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:18:30,546 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:18:30,546 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:18:30,547 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:18:30,554 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:18:30,564 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:18:30.564064 [2023-09-07 18:18:30,815 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:30,911 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:18:30,968 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:18:30,971 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:18:30,982 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:18:30,983 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694110710.909011, 'te': 1694110710.9182503, 'initial_status': True} ====================================================================================================  [2023-09-07 18:18:30,984 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:18:30.984866 [2023-09-07 18:18:30,989 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:18:30,990 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:18:34,524 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 18:18:34,709 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:18:34,710 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:18:34,710 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:18:34,710 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 18:18:34,887 INFO][11][wan-tasty-kitalpha][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 18:18:34,887 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 18:18:34,889 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 18:18:34,894 ERROR][11][wan-tasty-kitalpha][T_1] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:18:34,895 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:18:38,699 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110716, schain_finish_ts: 1694108304 [2023-09-07 18:18:39,844 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:18:39,844 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:18:39,844 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:18:39,844 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.task:40 - Task free-basic-rosy-SKALE-config finished with None [2023-09-07 18:18:39,845 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.task:42 - Running task free-basic-rosy-SKALE-config [2023-09-07 18:18:39,846 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.task:40 - Task free-basic-rosy-SKALE-skaled finished with None [2023-09-07 18:18:39,848 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.task:42 - Running task free-basic-rosy-SKALE-skaled [2023-09-07 18:18:39,849 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 25 [2023-09-07 18:18:39,881 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.firewall.utils:42 - Creating rule controller for free-basic-rosy-SKALE [2023-09-07 18:18:40,286 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:40,493 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:40,602 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': False, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:18:40,603 INFO][30622][free-basic-rosy-SKALE][T_0] - tools.notifications.messages:129 - Sending checks notification with state [('blocks', False), ('config', True), ('config_updated', True), ('exit_code_ok', True), ('exit_zero', False), ('firewall_rules', True), ('ima_container', True), ('process', True), ('rotation_id_updated', True), ('rpc', True), ('skaled_container', True), ('upstream_exists', True), ('volume', True)] [2023-09-07 18:18:40,605 INFO][30622][free-basic-rosy-SKALE][T_0] - tools.notifications.messages:134 - Saving new checks state 1 [('blocks', False), ('config', True), ('config_updated', True), ('exit_code_ok', True), ('exit_zero', False), ('firewall_rules', True), ('ima_container', True), ('process', True), ('rotation_id_updated', True), ('rpc', True), ('skaled_container', True), ('upstream_exists', True), ('volume', True)] [2023-09-07 18:18:40,606 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/free-basic-rosy-SKALE/schain_free-basic-rosy-SKALE_3_1694110601.json [2023-09-07 18:18:40,606 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': False, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:18:40,606 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:18:40,607 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:18:40,607 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:18:40,613 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:18:40,614 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:152 - Changing monitor_last_seen for free-basic-rosy-SKALE to 2023-09-07 18:18:40.614944 [2023-09-07 18:18:40,829 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:40,860 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:167 - Changing restart count for free-basic-rosy-SKALE to 0 [2023-09-07 18:18:40,913 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:127 - Changing first_run for free-basic-rosy-SKALE to False [2023-09-07 18:18:40,917 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:142 - Changing new_schain for free-basic-rosy-SKALE to False [2023-09-07 18:18:40,936 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:18:40,937 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - free-basic-rosy-SKALE reset_restart_counter: {'ts': 1694110720.8576303, 'te': 1694110720.8707788, 'initial_status': True} ====================================================================================================  [2023-09-07 18:18:40,939 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:152 - Changing monitor_last_seen for free-basic-rosy-SKALE to 2023-09-07 18:18:40.939898 [2023-09-07 18:18:40,948 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:18:40,949 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:18:41,693 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:18:41,693 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:18:41,695 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:152 - Changing monitor_last_seen for free-basic-rosy-SKALE to 2023-09-07 18:18:41.695580 [2023-09-07 18:18:41,703 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.config.main:65 - Going to change config_version for free-basic-rosy-SKALE: regression-2.2 -> regression-2.2 [2023-09-07 18:18:41,703 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:162 - Changing config_version for free-basic-rosy-SKALE to regression-2.2 [2023-09-07 18:18:41,709 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:190 - Changing sync_config_run for free-basic-rosy-SKALE to False [2023-09-07 18:18:41,713 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - free-basic-rosy-SKALE reset_config_record: {'ts': 1694110721.7019138, 'te': 1694110721.7135656, 'initial_status': True} ====================================================================================================  [2023-09-07 18:18:41,715 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:152 - Changing monitor_last_seen for free-basic-rosy-SKALE to 2023-09-07 18:18:41.715746 [2023-09-07 18:18:41,721 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:18:41,722 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:18:44,662 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110716, schain_finish_ts: 1692377364 [2023-09-07 18:18:45,797 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:18:45,798 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:18:45,798 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:18:45,798 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:18:45,798 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:18:45,798 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:18:45,800 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:18:45,801 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 18:18:45,826 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:18:46,049 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:46,344 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:46,456 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:18:46,458 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2570 [2023-09-07 18:18:46,459 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:18:46,459 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:18:46,459 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:18:46,460 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:18:46,460 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:18:46,472 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:18:46,476 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:18:46.476892 [2023-09-07 18:18:46,683 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:46,716 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:18:46,722 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:159 - sChain: curly-red-alterf. Node id on chain is 15; Node id on contract is 16 [2023-09-07 18:18:46,723 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:162 - sChain: curly-red-alterf. DKG timeout is 1800 [2023-09-07 18:18:46,761 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:18:46,764 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:18:46,772 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:18:46,773 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694110726.7143672, 'te': 1694110726.7219887, 'initial_status': True} ====================================================================================================  [2023-09-07 18:18:46,775 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:18:46.775670 [2023-09-07 18:18:46,780 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:18:46,781 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:18:47,633 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:18:47,633 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:18:47,635 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:18:47.635432 [2023-09-07 18:18:47,643 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:18:47,643 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:18:47,649 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:18:47,654 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694110727.641699, 'te': 1694110727.653992, 'initial_status': True} ====================================================================================================  [2023-09-07 18:18:47,656 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:18:47.656469 [2023-09-07 18:18:47,660 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:18:47,661 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:18:51,617 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110728, schain_finish_ts: 0 [2023-09-07 18:18:52,622 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110728, schain_finish_ts: 1693742304 [2023-09-07 18:18:52,748 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:18:52,749 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:18:52,750 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 18:18:53,756 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:18:53,757 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:18:53,757 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:18:53,757 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-config finished with None [2023-09-07 18:18:53,757 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-config [2023-09-07 18:18:53,758 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:18:53,758 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:18:53,759 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 18:18:53,795 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:18:54,072 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:54,263 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:54,361 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:18:54,362 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2593 [2023-09-07 18:18:54,363 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:18:54,363 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:18:54,363 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:18:54,363 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:18:54,364 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:18:54,370 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:18:54,372 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:18:54.372096 [2023-09-07 18:18:54,681 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:54,708 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:18:54,736 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:18:54,739 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:18:54,746 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:18:54,747 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694110734.7073047, 'te': 1694110734.7131567, 'initial_status': True} ====================================================================================================  [2023-09-07 18:18:54,748 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:18:54.748875 [2023-09-07 18:18:54,753 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:18:54,754 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:18:55,659 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 18:18:55,659 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:18:55,662 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:18:55.662358 [2023-09-07 18:18:55,667 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:18:55,998 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110728, schain_finish_ts: 1693412520 [2023-09-07 18:18:57,137 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:18:57,138 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:18:57,138 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:18:57,138 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:18:57,138 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:18:57,139 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:18:57,140 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:18:57,141 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 24 [2023-09-07 18:18:57,163 INFO][9][howling-hoedus][T_0] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:18:57,383 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:57,687 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:57,791 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:18:57,792 INFO][9][howling-hoedus][T_0] - tools.notifications.messages:142 - Saving new checks count 2589 [2023-09-07 18:18:57,792 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:18:57,792 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:18:57,793 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:18:57,793 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:18:57,793 INFO][9][howling-hoedus][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:18:57,799 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:18:57,801 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:18:57.801258 [2023-09-07 18:18:58,028 INFO][9][howling-hoedus][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:18:58,054 INFO][9][howling-hoedus][T_0] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:18:58,087 INFO][9][howling-hoedus][T_0] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:18:58,089 INFO][9][howling-hoedus][T_0] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:18:58,096 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:18:58,096 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694110738.0527437, 'te': 1694110738.0592868, 'initial_status': True} ====================================================================================================  [2023-09-07 18:18:58,098 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:18:58.098193 [2023-09-07 18:18:58,102 INFO][9][howling-hoedus][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:18:58,103 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:18:58,968 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:18:58,968 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:18:58,970 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:18:58.970603 [2023-09-07 18:18:58,976 INFO][9][howling-hoedus][T_1] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:18:58,977 INFO][9][howling-hoedus][T_1] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:18:58,983 INFO][9][howling-hoedus][T_1] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:18:58,988 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694110738.975421, 'te': 1694110738.9880714, 'initial_status': True} ====================================================================================================  [2023-09-07 18:18:58,990 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:18:58.990020 [2023-09-07 18:18:58,995 INFO][9][howling-hoedus][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:18:58,995 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:19:01,931 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110740, schain_finish_ts: 1692637668 [2023-09-07 18:19:03,069 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:19:03,070 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:19:03,070 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:19:03,070 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:19:03,071 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:19:03,071 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:19:03,073 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:19:03,076 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 27 [2023-09-07 18:19:03,102 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:19:03,456 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:03,660 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:03,769 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:19:03,770 INFO][7][portly-passionate-sirius][T_0] - tools.notifications.messages:142 - Saving new checks count 2586 [2023-09-07 18:19:03,771 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:19:03,771 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:19:03,771 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:19:03,771 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:19:03,772 INFO][7][portly-passionate-sirius][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:19:03,778 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:19:03,780 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:19:03.780139 [2023-09-07 18:19:04,007 INFO][7][portly-passionate-sirius][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:04,048 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:19:04,104 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:19:04,106 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:19:04,113 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:19:04,113 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694110744.0434287, 'te': 1694110744.0583463, 'initial_status': True} ====================================================================================================  [2023-09-07 18:19:04,115 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:19:04.115428 [2023-09-07 18:19:04,120 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:19:04,120 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:19:04,922 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:19:04,922 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:19:04,925 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:19:04.925149 [2023-09-07 18:19:04,933 INFO][7][portly-passionate-sirius][T_1] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:19:04,934 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:19:04,941 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:19:04,947 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694110744.9314492, 'te': 1694110744.947265, 'initial_status': True} ====================================================================================================  [2023-09-07 18:19:04,949 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:19:04.949761 [2023-09-07 18:19:04,955 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:19:04,955 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:19:05,791 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110740, schain_finish_ts: 1694108304 [2023-09-07 18:19:05,814 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110740, schain_finish_ts: 1693742004 [2023-09-07 18:19:06,153 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16'}} [2023-09-07 18:19:06,338 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:19:06,338 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 18:19:06,339 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.client:280 - sChain: curly-red-alterf. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16 [2023-09-07 18:19:06,339 INFO][8][curly-red-alterf][T_1] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:13761038544475430444871957621815481511415244541540880574831932689051904:NODE_ID:15:DKG_ID:16', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 18:19:06,528 INFO][8][curly-red-alterf][T_1] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 18:19:06,528 INFO][8][curly-red-alterf][T_1] - core.schains.dkg.main:181 - sChain curly-red-alterf DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 18:19:06,530 INFO][8][curly-red-alterf][T_1] - web.models.schain:118 - Changing DKG status for curly-red-alterf to KEY_GENERATION_ERROR [2023-09-07 18:19:06,538 ERROR][8][curly-red-alterf][T_1] - core.schains.task:26 - Task curly-red-alterf-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:19:06,539 INFO][8][curly-red-alterf][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:19:06,727 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110740, schain_finish_ts: 1692377364 [2023-09-07 18:19:06,914 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:19:06,914 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:19:06,914 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:19:06,914 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.task:40 - Task free-basic-rosy-SKALE-config finished with None [2023-09-07 18:19:06,915 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.task:42 - Running task free-basic-rosy-SKALE-config [2023-09-07 18:19:06,915 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.task:40 - Task free-basic-rosy-SKALE-skaled finished with None [2023-09-07 18:19:06,918 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.task:42 - Running task free-basic-rosy-SKALE-skaled [2023-09-07 18:19:06,920 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 18:19:06,934 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:19:06,934 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:19:06,934 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:19:06,935 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:19:06,935 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:19:06,936 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 30 [2023-09-07 18:19:06,948 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.firewall.utils:42 - Creating rule controller for free-basic-rosy-SKALE [2023-09-07 18:19:06,969 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:19:07,209 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:07,360 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:07,541 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:07,692 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': False, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:19:07,694 INFO][30622][free-basic-rosy-SKALE][T_1] - tools.notifications.messages:142 - Saving new checks count 2 [2023-09-07 18:19:07,695 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/free-basic-rosy-SKALE/schain_free-basic-rosy-SKALE_3_1694110601.json [2023-09-07 18:19:07,695 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': False, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:19:07,695 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:19:07,696 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:19:07,696 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:19:07,703 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:19:07,705 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:152 - Changing monitor_last_seen for free-basic-rosy-SKALE to 2023-09-07 18:19:07.705302 [2023-09-07 18:19:07,732 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:07,847 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:19:07,848 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2584 [2023-09-07 18:19:07,849 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:19:07,849 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:19:07,849 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:19:07,850 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:19:07,850 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:19:07,857 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:19:07,859 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:19:07.859588 [2023-09-07 18:19:07,859 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:19:07,860 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:19:07,860 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:19:07,861 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:19:07,861 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:19:07,862 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:19:07,863 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:19:07,864 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 18:19:07,886 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:19:08,079 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:08,109 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:167 - Changing restart count for free-basic-rosy-SKALE to 0 [2023-09-07 18:19:08,160 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:127 - Changing first_run for free-basic-rosy-SKALE to False [2023-09-07 18:19:08,167 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:142 - Changing new_schain for free-basic-rosy-SKALE to False [2023-09-07 18:19:08,176 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:19:08,176 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - free-basic-rosy-SKALE reset_restart_counter: {'ts': 1694110748.1069674, 'te': 1694110748.1161377, 'initial_status': True} ====================================================================================================  [2023-09-07 18:19:08,178 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:152 - Changing monitor_last_seen for free-basic-rosy-SKALE to 2023-09-07 18:19:08.178370 [2023-09-07 18:19:08,183 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:19:08,184 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:19:08,361 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:08,401 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:19:08,450 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:19:08,453 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:19:08,463 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:19:08,464 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694110748.3983855, 'te': 1694110748.4099233, 'initial_status': True} ====================================================================================================  [2023-09-07 18:19:08,466 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:19:08.465988 [2023-09-07 18:19:08,470 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:19:08,471 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:19:08,603 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:08,791 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:19:08,791 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:19:08,793 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:152 - Changing monitor_last_seen for free-basic-rosy-SKALE to 2023-09-07 18:19:08.793405 [2023-09-07 18:19:08,800 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.config.main:65 - Going to change config_version for free-basic-rosy-SKALE: regression-2.2 -> regression-2.2 [2023-09-07 18:19:08,800 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:162 - Changing config_version for free-basic-rosy-SKALE to regression-2.2 [2023-09-07 18:19:08,806 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:190 - Changing sync_config_run for free-basic-rosy-SKALE to False [2023-09-07 18:19:08,810 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - free-basic-rosy-SKALE reset_config_record: {'ts': 1694110748.7988296, 'te': 1694110748.8104224, 'initial_status': True} ====================================================================================================  [2023-09-07 18:19:08,812 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:152 - Changing monitor_last_seen for free-basic-rosy-SKALE to 2023-09-07 18:19:08.812463 [2023-09-07 18:19:08,816 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:19:08,817 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:19:08,831 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:08,959 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:19:08,960 INFO][10][green-giddy-denebola][T_0] - tools.notifications.messages:142 - Saving new checks count 2571 [2023-09-07 18:19:08,960 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:19:08,960 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:19:08,961 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:19:08,961 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:19:08,962 INFO][10][green-giddy-denebola][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:19:08,968 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:19:08,970 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:19:08.970916 [2023-09-07 18:19:09,180 INFO][10][green-giddy-denebola][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:09,218 INFO][10][green-giddy-denebola][T_0] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:19:09,245 INFO][10][green-giddy-denebola][T_0] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:19:09,247 INFO][10][green-giddy-denebola][T_0] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:19:09,253 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:19:09,254 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694110749.210581, 'te': 1694110749.2233496, 'initial_status': True} ====================================================================================================  [2023-09-07 18:19:09,255 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:19:09.255874 [2023-09-07 18:19:09,260 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:19:09,260 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:19:09,641 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:19:09,641 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:19:09,644 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:19:09.644062 [2023-09-07 18:19:09,652 INFO][10][green-giddy-denebola][T_1] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:19:09,652 INFO][10][green-giddy-denebola][T_1] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:19:09,657 INFO][10][green-giddy-denebola][T_1] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:19:09,661 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694110749.6506214, 'te': 1694110749.6614907, 'initial_status': True} ====================================================================================================  [2023-09-07 18:19:09,663 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:19:09.663237 [2023-09-07 18:19:09,666 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:19:09,667 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:19:14,272 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:159 - sChain: wan-tasty-kitalpha. Node id on chain is 4; Node id on contract is 16 [2023-09-07 18:19:14,272 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:162 - sChain: wan-tasty-kitalpha. DKG timeout is 1800 [2023-09-07 18:19:17,685 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110740, schain_finish_ts: 0 [2023-09-07 18:19:18,807 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:19:18,807 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:19:18,807 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 18:19:22,065 INFO][9][howling-hoedus][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110740, schain_finish_ts: 1693412520 [2023-09-07 18:19:23,177 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:19:23,177 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:19:23,178 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:19:23,178 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-config finished with None [2023-09-07 18:19:23,178 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-config [2023-09-07 18:19:23,179 INFO][9][howling-hoedus][MainThread] - core.schains.task:40 - Task howling-hoedus-skaled finished with None [2023-09-07 18:19:23,181 INFO][9][howling-hoedus][MainThread] - core.schains.task:42 - Running task howling-hoedus-skaled [2023-09-07 18:19:23,186 INFO][9][howling-hoedus][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 39 [2023-09-07 18:19:23,207 INFO][9][howling-hoedus][T_1] - core.schains.firewall.utils:42 - Creating rule controller for howling-hoedus [2023-09-07 18:19:23,484 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:23,656 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:23,755 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:19:23,755 INFO][9][howling-hoedus][T_1] - tools.notifications.messages:142 - Saving new checks count 2590 [2023-09-07 18:19:23,756 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/howling-hoedus/schain_howling-hoedus_25_1693506090.json [2023-09-07 18:19:23,756 INFO][9][howling-hoedus][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:19:23,757 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:19:23,757 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:19:23,757 INFO][9][howling-hoedus][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:19:23,764 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:19:23,765 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:19:23.765916 [2023-09-07 18:19:24,057 INFO][9][howling-hoedus][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:24,090 INFO][9][howling-hoedus][T_1] - web.models.schain:167 - Changing restart count for howling-hoedus to 0 [2023-09-07 18:19:24,126 INFO][9][howling-hoedus][T_1] - web.models.schain:127 - Changing first_run for howling-hoedus to False [2023-09-07 18:19:24,128 INFO][9][howling-hoedus][T_1] - web.models.schain:142 - Changing new_schain for howling-hoedus to False [2023-09-07 18:19:24,135 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:19:24,136 INFO][9][howling-hoedus][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_restart_counter: {'ts': 1694110764.0862908, 'te': 1694110764.1005754, 'initial_status': True} ====================================================================================================  [2023-09-07 18:19:24,137 INFO][9][howling-hoedus][T_1] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:19:24.137860 [2023-09-07 18:19:24,142 INFO][9][howling-hoedus][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:19:24,143 INFO][9][howling-hoedus][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:19:24,721 INFO][11][wan-tasty-kitalpha][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110740, schain_finish_ts: 1693742304 [2023-09-07 18:19:25,008 INFO][9][howling-hoedus][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:19:25,008 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:19:25,011 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:19:25.011913 [2023-09-07 18:19:25,024 INFO][9][howling-hoedus][T_0] - core.schains.config.main:65 - Going to change config_version for howling-hoedus: regression-2.2 -> regression-2.2 [2023-09-07 18:19:25,024 INFO][9][howling-hoedus][T_0] - web.models.schain:162 - Changing config_version for howling-hoedus to regression-2.2 [2023-09-07 18:19:25,033 INFO][9][howling-hoedus][T_0] - web.models.schain:190 - Changing sync_config_run for howling-hoedus to False [2023-09-07 18:19:25,041 INFO][9][howling-hoedus][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - howling-hoedus reset_config_record: {'ts': 1694110765.019063, 'te': 1694110765.0411026, 'initial_status': True} ====================================================================================================  [2023-09-07 18:19:25,044 INFO][9][howling-hoedus][T_0] - web.models.schain:152 - Changing monitor_last_seen for howling-hoedus to 2023-09-07 18:19:25.044741 [2023-09-07 18:19:25,050 INFO][9][howling-hoedus][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:19:25,050 INFO][9][howling-hoedus][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:19:25,868 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:19:25,868 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:19:25,869 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:19:25,869 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:40 - Task wan-tasty-kitalpha-skaled finished with None [2023-09-07 18:19:25,869 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.task:42 - Running task wan-tasty-kitalpha-skaled [2023-09-07 18:19:25,869 INFO][11][wan-tasty-kitalpha][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 38 [2023-09-07 18:19:25,889 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.utils:42 - Creating rule controller for wan-tasty-kitalpha [2023-09-07 18:19:26,133 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:26,396 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:26,538 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:19:26,539 INFO][11][wan-tasty-kitalpha][T_1] - tools.notifications.messages:142 - Saving new checks count 2594 [2023-09-07 18:19:26,542 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/wan-tasty-kitalpha/schain_wan-tasty-kitalpha_9_1693506035.json [2023-09-07 18:19:26,544 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:19:26,544 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:19:26,545 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:19:26,545 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:19:26,550 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:19:26,552 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:19:26.552866 [2023-09-07 18:19:26,772 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:26,807 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:167 - Changing restart count for wan-tasty-kitalpha to 0 [2023-09-07 18:19:26,843 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:127 - Changing first_run for wan-tasty-kitalpha to False [2023-09-07 18:19:26,845 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:142 - Changing new_schain for wan-tasty-kitalpha to False [2023-09-07 18:19:26,854 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:19:26,855 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - wan-tasty-kitalpha reset_restart_counter: {'ts': 1694110766.805308, 'te': 1694110766.8135495, 'initial_status': True} ====================================================================================================  [2023-09-07 18:19:26,857 INFO][11][wan-tasty-kitalpha][T_1] - web.models.schain:152 - Changing monitor_last_seen for wan-tasty-kitalpha to 2023-09-07 18:19:26.857203 [2023-09-07 18:19:26,862 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:19:26,863 INFO][11][wan-tasty-kitalpha][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:19:31,004 INFO][7][portly-passionate-sirius][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110764, schain_finish_ts: 1692637668 [2023-09-07 18:19:32,144 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:19:32,144 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:19:32,144 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:19:32,144 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-config finished with None [2023-09-07 18:19:32,145 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-config [2023-09-07 18:19:32,145 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:40 - Task portly-passionate-sirius-skaled finished with None [2023-09-07 18:19:32,146 INFO][7][portly-passionate-sirius][MainThread] - core.schains.task:42 - Running task portly-passionate-sirius-skaled [2023-09-07 18:19:32,148 INFO][7][portly-passionate-sirius][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 36 [2023-09-07 18:19:32,180 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.utils:42 - Creating rule controller for portly-passionate-sirius [2023-09-07 18:19:32,456 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:32,631 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:32,730 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:19:32,731 INFO][7][portly-passionate-sirius][T_1] - tools.notifications.messages:142 - Saving new checks count 2587 [2023-09-07 18:19:32,732 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/portly-passionate-sirius/schain_portly-passionate-sirius_24_1693506079.json [2023-09-07 18:19:32,732 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:19:32,733 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:19:32,733 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:19:32,733 INFO][7][portly-passionate-sirius][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:19:32,740 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:19:32,742 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:19:32.742791 [2023-09-07 18:19:33,110 INFO][7][portly-passionate-sirius][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:33,148 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:167 - Changing restart count for portly-passionate-sirius to 0 [2023-09-07 18:19:33,186 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:127 - Changing first_run for portly-passionate-sirius to False [2023-09-07 18:19:33,188 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:142 - Changing new_schain for portly-passionate-sirius to False [2023-09-07 18:19:33,200 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:19:33,201 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_restart_counter: {'ts': 1694110773.1470003, 'te': 1694110773.154608, 'initial_status': True} ====================================================================================================  [2023-09-07 18:19:33,203 INFO][7][portly-passionate-sirius][T_1] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:19:33.202980 [2023-09-07 18:19:33,209 INFO][7][portly-passionate-sirius][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:19:33,209 INFO][7][portly-passionate-sirius][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:19:33,425 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'getBLSPublicKeyShare', 'params': {'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14'}} [2023-09-07 18:19:33,602 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'getBLSPublicKeyShareImpl failed:readFromDb:Data wi...', 'status': -7}} [2023-09-07 18:19:33,603 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:269 - No bls key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14, getBLSPublicKeyShareImpl failed:readFromDb:Data with this name does not exist: BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:19:33,603 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.client:280 - sChain: wan-tasty-kitalpha. DKGClient is going to create BLS private key with name BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14 [2023-09-07 18:19:33,603 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:49 - Send request: {'id': 0, 'jsonrpc': '2.0', 'method': 'createBLSPrivateKeyV2', 'params': {'polyName': 'POLY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'blsKeyName': 'BLS_KEY:SCHAIN_ID:9136867521045401666888946550962317902651028791413595327844817200074748:NODE_ID:4:DKG_ID:14', 'ethKeyName': '[SGX_KEY]', 'secretShare': '0000000000000000', 'n': 16, 't': 11}} [2023-09-07 18:19:33,789 INFO][11][wan-tasty-kitalpha][T_0] - sgx.utils:55 - Response received: {'id': 0, 'jsonrpc': '2.0', 'result': {'errorMessage': 'createBLSPrivateKeyV2Impl failed:createBLSPrivateK...', 'status': -52}} [2023-09-07 18:19:33,790 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.dkg.main:181 - sChain wan-tasty-kitalpha DKG failed during key generation, err createBLSPrivateKeyV2Impl failed:createBLSPrivateKeyV2Impl:Invalid secret share length [2023-09-07 18:19:33,792 INFO][11][wan-tasty-kitalpha][T_0] - web.models.schain:118 - Changing DKG status for wan-tasty-kitalpha to KEY_GENERATION_ERROR [2023-09-07 18:19:33,796 ERROR][11][wan-tasty-kitalpha][T_0] - core.schains.task:26 - Task wan-tasty-kitalpha-config failed with DKG failed Traceback (most recent call last): File "/usr/src/admin/core/schains/task.py", line 24, in run self.action() File "/usr/src/admin/core/schains/monitor/main.py", line 105, in run_config_pipeline mon.run() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 48, in run self.execute() File "/usr/src/admin/core/schains/monitor/config_monitor.py", line 59, in execute self.am.dkg() File "/usr/src/admin/core/schains/monitor/action.py", line 96, in _monitor_block initial_status = f(self, *args, **kwargs) File "/usr/src/admin/core/schains/monitor/action.py", line 183, in dkg raise DkgError('DKG failed') core.schains.dkg.client.DkgError: DKG failed [2023-09-07 18:19:33,797 INFO][11][wan-tasty-kitalpha][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:19:33,938 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:19:33,938 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:19:33,944 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:19:33.944940 [2023-09-07 18:19:33,952 INFO][7][portly-passionate-sirius][T_0] - core.schains.config.main:65 - Going to change config_version for portly-passionate-sirius: regression-2.2 -> regression-2.2 [2023-09-07 18:19:33,952 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:162 - Changing config_version for portly-passionate-sirius to regression-2.2 [2023-09-07 18:19:33,959 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:190 - Changing sync_config_run for portly-passionate-sirius to False [2023-09-07 18:19:33,966 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - portly-passionate-sirius reset_config_record: {'ts': 1694110773.951071, 'te': 1694110773.9667127, 'initial_status': True} ====================================================================================================  [2023-09-07 18:19:33,969 INFO][7][portly-passionate-sirius][T_0] - web.models.schain:152 - Changing monitor_last_seen for portly-passionate-sirius to 2023-09-07 18:19:33.969034 [2023-09-07 18:19:33,974 INFO][7][portly-passionate-sirius][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:19:33,974 INFO][7][portly-passionate-sirius][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:19:37,830 INFO][30622][free-basic-rosy-SKALE][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110764, schain_finish_ts: 1694108304 [2023-09-07 18:19:37,907 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110764, schain_finish_ts: 1693742004 [2023-09-07 18:19:38,938 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:19:38,939 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:19:38,939 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:19:38,939 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.task:40 - Task free-basic-rosy-SKALE-config finished with None [2023-09-07 18:19:38,939 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.task:42 - Running task free-basic-rosy-SKALE-config [2023-09-07 18:19:38,940 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.task:40 - Task free-basic-rosy-SKALE-skaled finished with None [2023-09-07 18:19:38,941 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.task:42 - Running task free-basic-rosy-SKALE-skaled [2023-09-07 18:19:38,944 INFO][30622][free-basic-rosy-SKALE][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 35 [2023-09-07 18:19:38,976 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.firewall.utils:42 - Creating rule controller for free-basic-rosy-SKALE [2023-09-07 18:19:39,043 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:19:39,043 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:19:39,044 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:19:39,044 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-config finished with None [2023-09-07 18:19:39,044 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-config [2023-09-07 18:19:39,045 INFO][8][curly-red-alterf][MainThread] - core.schains.task:40 - Task curly-red-alterf-skaled finished with None [2023-09-07 18:19:39,046 INFO][8][curly-red-alterf][MainThread] - core.schains.task:42 - Running task curly-red-alterf-skaled [2023-09-07 18:19:39,048 INFO][8][curly-red-alterf][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 20 [2023-09-07 18:19:39,073 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.utils:42 - Creating rule controller for curly-red-alterf [2023-09-07 18:19:39,302 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:39,711 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:39,900 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:40,007 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': False, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:19:40,008 INFO][30622][free-basic-rosy-SKALE][T_0] - tools.notifications.messages:142 - Saving new checks count 3 [2023-09-07 18:19:40,009 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/free-basic-rosy-SKALE/schain_free-basic-rosy-SKALE_3_1694110601.json [2023-09-07 18:19:40,009 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': False, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:19:40,009 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:19:40,009 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:19:40,010 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:19:40,018 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:19:40,020 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:152 - Changing monitor_last_seen for free-basic-rosy-SKALE to 2023-09-07 18:19:40.020265 [2023-09-07 18:19:40,088 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:40,232 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:19:40,233 INFO][8][curly-red-alterf][T_0] - tools.notifications.messages:142 - Saving new checks count 2585 [2023-09-07 18:19:40,234 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/curly-red-alterf/schain_curly-red-alterf_11_1693506042.json [2023-09-07 18:19:40,234 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:19:40,235 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:19:40,235 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:19:40,235 INFO][8][curly-red-alterf][T_0] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:19:40,244 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:19:40,246 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:19:40.246379 [2023-09-07 18:19:40,293 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:40,323 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:167 - Changing restart count for free-basic-rosy-SKALE to 0 [2023-09-07 18:19:40,353 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:127 - Changing first_run for free-basic-rosy-SKALE to False [2023-09-07 18:19:40,355 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:142 - Changing new_schain for free-basic-rosy-SKALE to False [2023-09-07 18:19:40,363 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:19:40,364 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - free-basic-rosy-SKALE reset_restart_counter: {'ts': 1694110780.3215625, 'te': 1694110780.3305678, 'initial_status': True} ====================================================================================================  [2023-09-07 18:19:40,366 INFO][30622][free-basic-rosy-SKALE][T_0] - web.models.schain:152 - Changing monitor_last_seen for free-basic-rosy-SKALE to 2023-09-07 18:19:40.366314 [2023-09-07 18:19:40,372 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:19:40,376 INFO][30622][free-basic-rosy-SKALE][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:19:40,555 INFO][8][curly-red-alterf][T_0] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:40,585 INFO][8][curly-red-alterf][T_0] - web.models.schain:167 - Changing restart count for curly-red-alterf to 0 [2023-09-07 18:19:40,619 INFO][8][curly-red-alterf][T_0] - web.models.schain:127 - Changing first_run for curly-red-alterf to False [2023-09-07 18:19:40,621 INFO][8][curly-red-alterf][T_0] - web.models.schain:142 - Changing new_schain for curly-red-alterf to False [2023-09-07 18:19:40,631 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:19:40,632 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - curly-red-alterf reset_restart_counter: {'ts': 1694110780.5832906, 'te': 1694110780.5907612, 'initial_status': True} ====================================================================================================  [2023-09-07 18:19:40,634 INFO][8][curly-red-alterf][T_0] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:19:40.634241 [2023-09-07 18:19:40,638 INFO][8][curly-red-alterf][T_0] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:19:40,639 INFO][8][curly-red-alterf][T_0] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:19:40,700 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:19:40,702 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:19:40,704 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:152 - Changing monitor_last_seen for free-basic-rosy-SKALE to 2023-09-07 18:19:40.704724 [2023-09-07 18:19:40,715 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.config.main:65 - Going to change config_version for free-basic-rosy-SKALE: regression-2.2 -> regression-2.2 [2023-09-07 18:19:40,717 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:162 - Changing config_version for free-basic-rosy-SKALE to regression-2.2 [2023-09-07 18:19:40,726 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:190 - Changing sync_config_run for free-basic-rosy-SKALE to False [2023-09-07 18:19:40,733 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - free-basic-rosy-SKALE reset_config_record: {'ts': 1694110780.7136512, 'te': 1694110780.7330065, 'initial_status': True} ====================================================================================================  [2023-09-07 18:19:40,736 INFO][30622][free-basic-rosy-SKALE][T_1] - web.models.schain:152 - Changing monitor_last_seen for free-basic-rosy-SKALE to 2023-09-07 18:19:40.736412 [2023-09-07 18:19:40,742 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:19:40,743 INFO][30622][free-basic-rosy-SKALE][T_1] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:19:40,864 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': False, 'external_state': True, 'upstream_config': False} [2023-09-07 18:19:40,864 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:19:40,866 INFO][8][curly-red-alterf][T_1] - web.models.schain:152 - Changing monitor_last_seen for curly-red-alterf to 2023-09-07 18:19:40.866890 [2023-09-07 18:19:40,874 INFO][8][curly-red-alterf][T_1] - core.schains.monitor.action:168 - Running safe_run_dkg [2023-09-07 18:19:41,578 INFO][1][MainProcess][MainThread] - core.schains.process_manager:57 - Process manager started [2023-09-07 18:19:41,794 INFO][1][MainProcess][MainThread] - core.schains.notifications:35 - Node account has 407295140193863165 WEI [2023-09-07 18:19:41,796 INFO][1][MainProcess][MainThread] - tools.notifications.messages:188 - Saving new balance state 3023 1 [2023-09-07 18:19:41,796 INFO][1][MainProcess][MainThread] - core.schains.process_manager:95 - Fetching schains to monitor... [2023-09-07 18:19:44,698 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:40 - Collecting rotation history for submissive-well-off-rotanev... [2023-09-07 18:19:45,795 INFO][10][green-giddy-denebola][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110764, schain_finish_ts: 1692377364 [2023-09-07 18:19:45,840 INFO][24912][submissive-well-off-rotanev][T_0] - skale.schain_config.rotation_history:50 - Rotation data for submissive-well-off-rotanev: Rotation(leaving_node_id=65, new_node_id=65, freeze_until=1694108100, rotation_counter=8) [2023-09-07 18:19:46,906 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version regression-2.2, stream_version regression-2.2 [2023-09-07 18:19:46,906 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:19:46,906 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:207 - Adding skaled task to the pool [2023-09-07 18:19:46,907 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-config finished with None [2023-09-07 18:19:46,907 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-config [2023-09-07 18:19:46,907 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:40 - Task green-giddy-denebola-skaled finished with None [2023-09-07 18:19:46,908 INFO][10][green-giddy-denebola][MainThread] - core.schains.task:42 - Running task green-giddy-denebola-skaled [2023-09-07 18:19:46,909 INFO][10][green-giddy-denebola][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 32 [2023-09-07 18:19:46,931 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.utils:42 - Creating rule controller for green-giddy-denebola [2023-09-07 18:19:47,270 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:47,389 INFO][1][MainProcess][MainThread] - core.schains.process_manager:109 - Get leaving_history for node ... [2023-09-07 18:19:47,455 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:47,574 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:138 - Skaled checks: {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:19:47,575 INFO][10][green-giddy-denebola][T_1] - tools.notifications.messages:142 - Saving new checks count 2572 [2023-09-07 18:19:47,575 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:141 - Upstream config /skale_node_data/schains/green-giddy-denebola/schain_green-giddy-denebola_13_1693506053.json [2023-09-07 18:19:47,575 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.main:142 - Status dict {'blocks': True, 'config': True, 'config_updated': True, 'exit_code_ok': True, 'exit_zero': False, 'firewall_rules': True, 'ima_container': True, 'process': True, 'rotation_id_updated': True, 'rpc': True, 'skaled_container': True, 'upstream_exists': True, 'volume': True} [2023-09-07 18:19:47,576 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:273 - Choosing skaled monitor [2023-09-07 18:19:47,576 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:19:47,576 INFO][10][green-giddy-denebola][T_1] - core.schains.skaled_status:99 - skaled status file: { "subsystemRunning": { "SnapshotDownloader": false, "WaitingForTimestamp": false, "Blockchain": true, "Rpc": true }, "exitState": { "ClearDataDir": false, "StartAgain": true, "StartFromSnapshot": false, "ExitTimeReached": false } } [2023-09-07 18:19:47,582 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:51 - Skaled monitor type RegularSkaledMonitor starting [2023-09-07 18:19:47,584 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:19:47.584635 [2023-09-07 18:19:47,757 INFO][10][green-giddy-denebola][T_1] - core.schains.firewall.rule_controller:192 - Rules status: missing rules 0, redundant rules: 0 [2023-09-07 18:19:47,782 INFO][10][green-giddy-denebola][T_1] - web.models.schain:167 - Changing restart count for green-giddy-denebola to 0 [2023-09-07 18:19:47,809 INFO][10][green-giddy-denebola][T_1] - web.models.schain:127 - Changing first_run for green-giddy-denebola to False [2023-09-07 18:19:47,811 INFO][10][green-giddy-denebola][T_1] - web.models.schain:142 - Changing new_schain for green-giddy-denebola to False [2023-09-07 18:19:47,818 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:119 - restart_count - 0, failed_rpc_count - 0 [2023-09-07 18:19:47,818 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_restart_counter: {'ts': 1694110787.7804775, 'te': 1694110787.7863274, 'initial_status': True} ====================================================================================================  [2023-09-07 18:19:47,820 INFO][10][green-giddy-denebola][T_1] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:19:47.820618 [2023-09-07 18:19:47,825 INFO][10][green-giddy-denebola][T_1] - core.schains.monitor.skaled_monitor:57 - Skaled monitor type RegularSkaledMonitor finished [2023-09-07 18:19:47,826 INFO][10][green-giddy-denebola][T_1] - core.schains.task:27 - Sleeping after task execution for 2 [2023-09-07 18:19:48,678 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.main:103 - Config checks: {'config_dir': True, 'dkg': True, 'external_state': True, 'upstream_config': True} [2023-09-07 18:19:48,678 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:46 - Config monitor type RegularConfigMonitor starting [2023-09-07 18:19:48,680 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:19:48.680689 [2023-09-07 18:19:48,688 INFO][10][green-giddy-denebola][T_0] - core.schains.config.main:65 - Going to change config_version for green-giddy-denebola: regression-2.2 -> regression-2.2 [2023-09-07 18:19:48,688 INFO][10][green-giddy-denebola][T_0] - web.models.schain:162 - Changing config_version for green-giddy-denebola to regression-2.2 [2023-09-07 18:19:48,693 INFO][10][green-giddy-denebola][T_0] - web.models.schain:190 - Changing sync_config_run for green-giddy-denebola to False [2023-09-07 18:19:48,697 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.action:126 -  ==================================================================================================== Finished monitor runner - green-giddy-denebola reset_config_record: {'ts': 1694110788.686399, 'te': 1694110788.6976712, 'initial_status': True} ====================================================================================================  [2023-09-07 18:19:48,699 INFO][10][green-giddy-denebola][T_0] - web.models.schain:152 - Changing monitor_last_seen for green-giddy-denebola to 2023-09-07 18:19:48.699759 [2023-09-07 18:19:48,704 INFO][10][green-giddy-denebola][T_0] - core.schains.monitor.config_monitor:51 - Config monitor type RegularConfigMonitor finished [2023-09-07 18:19:48,704 INFO][10][green-giddy-denebola][T_0] - core.schains.task:27 - Sleeping after task execution for 3 [2023-09-07 18:19:49,346 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110764, schain_finish_ts: 1692377616 [2023-09-07 18:19:51,315 INFO][1][MainProcess][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110788, schain_finish_ts: 0 [2023-09-07 18:19:51,684 INFO][1][MainProcess][MainThread] - core.schains.process_manager:117 - Got leaving sChains for the node: [] [2023-09-07 18:19:51,684 INFO][1][MainProcess][MainThread] - core.schains.process_manager:101 -  ==================================================================================================== Monitoring sChains Node ID: 16 sChains on node: [{'name': 'portly-passionate-sirius', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666281384, 'startBlock': 7803380, 'deposit': 0, 'index': 1, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x0000017723ca2', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'curly-red-alterf', 'mainnetOwner': '0xCAc4Ef6B7eE25dC7Be99543E4750D9a51ea59c2e', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1676058936, 'startBlock': 8469646, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000001fe6cdfa', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'howling-hoedus', 'mainnetOwner': '0x1026467C91fdA40aCc7e6935d1E52f67274FD669', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1666181400, 'startBlock': 7796660, 'deposit': 0, 'index': 0, 'generation': 0, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0xe5a600f4c2608', 'multitransactionMode': False, 'thresholdEncryption': False, 'active': True}, {'name': 'green-giddy-denebola', 'mainnetOwner': '0x8e835d5d9E28ffD259E8Fa8988Cacfc333A582B4', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674565164, 'startBlock': 8368049, 'deposit': 0, 'index': 4, 'generation': 1, 'originator': '0xd2001DAb6898127Be2F167B548691C87251D13C3', 'chainId': '0x00000585eb4b1', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'wan-tasty-kitalpha', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1674568392, 'startBlock': 8368271, 'deposit': 0, 'index': 5, 'generation': 1, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x00000152e7be9', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'free-basic-rosy-SKALE', 'mainnetOwner': '0xb04a7f2Cd74Eb624Df4BD8bed8f7034B5Aed89C0', 'indexInOwnerList': 1, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1693996920, 'startBlock': 9646443, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000002bb90439', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}, {'name': 'submissive-well-off-rotanev', 'mainnetOwner': '0x487360568fE5Ee4AE5c75782D5F63e9935B0E512', 'indexInOwnerList': 0, 'partOfNode': 16, 'lifetime': 43200, 'startDate': 1692361104, 'startBlock': 9539053, 'deposit': 0, 'index': 7, 'generation': 2, 'originator': '0x0000000000000000000000000000000000000000', 'chainId': '0x000003791fa10', 'multitransactionMode': True, 'thresholdEncryption': False, 'active': True}] Number of sChains on node: 7 Empty sChain structs: 0 ====================================================================================================  [2023-09-07 18:19:52,054 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:19:52,054 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain portly-passionate-sirius - Process is running: PID = 7 [2023-09-07 18:19:52,427 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:19:52,428 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain curly-red-alterf - Process is running: PID = 8 [2023-09-07 18:19:52,797 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:19:52,798 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain howling-hoedus - Process is running: PID = 9 [2023-09-07 18:19:53,177 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:19:53,177 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain green-giddy-denebola - Process is running: PID = 10 [2023-09-07 18:19:53,547 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:19:53,548 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain wan-tasty-kitalpha - Process is running: PID = 11 [2023-09-07 18:19:53,920 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:19:53,921 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain free-basic-rosy-SKALE - Process is running: PID = 30622 [2023-09-07 18:19:54,297 INFO][1][MainProcess][MainThread] - core.schains.process_manager_helper:90 - dkg_timeout: 1800, TIMEOUT_COEFFICIENT: 2.2, allowed_diff: 3960 [2023-09-07 18:19:54,298 INFO][1][MainProcess][MainThread] - core.schains.process_manager:87 - sChain submissive-well-off-rotanev - Process is running: PID = 24912 [2023-09-07 18:19:54,298 INFO][1][MainProcess][MainThread] - core.schains.process_manager:88 - Process manager procedure finished [2023-09-07 18:19:54,298 INFO][1][MainProcess][MainThread] - __main__:65 - Sleeping for 90s after run_process_manager [2023-09-07 18:19:57,760 INFO][24912][submissive-well-off-rotanev][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110788, schain_finish_ts: 0 [2023-09-07 18:19:58,886 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:183 - sync_config_run False, config_version 0.0.0, stream_version regression-2.2 [2023-09-07 18:19:58,887 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:189 - Adding config task to the pool [2023-09-07 18:19:58,887 INFO][24912][submissive-well-off-rotanev][MainThread] - core.schains.monitor.main:157 - Monitor iteration completed, sleeping for 37 [2023-09-07 18:19:59,973 INFO][8][curly-red-alterf][MainThread] - skale.contracts.manager.node_rotation:115 - current_ts: 1694110788, schain_finish_ts: 1693742004