Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

indexer sync halt #62

Open
dnjef9duo opened this issue Jun 20, 2024 · 4 comments
Open

indexer sync halt #62

dnjef9duo opened this issue Jun 20, 2024 · 4 comments

Comments

@dnjef9duo
Copy link

dnjef9duo commented Jun 20, 2024 •
edited
Loading

i setup ton-indexer v3 for mainnet.

indexer halt everyday recently.

ton-indexer version
commit 72e5760db8b5ea4ce3371f2f583024cebf92c235 (HEAD -> master, origin/master, origin/HEAD)

my config for ton-indexer

# TON Indexer config
POSTGRES_HOST=postgres
POSTGRES_PORT=5432
POSTGRES_USER=postgres
POSTGRES_PASSWORD_FILE=private/postgres_password
POSTGRES_DBNAME=ton_index
POSTGRES_PUBLISH_PORT=5432

TON_INDEXER_API_ROOT_PATH=/api/v3
TON_INDEXER_API_PORT=81
TON_INDEXER_API_TITLE=TON Indexer
TON_INDEXER_WORKERS=4

TON_INDEXER_TON_HTTP_API_ENDPOINT=localhost

TON_WORKER_DBROOT=/var/ton-work/db/
TON_WORKER_FROM=38354000
TON_WORKER_MAX_PARALLEL_TASKS=1024
TON_WORKER_INSERT_BATCH_SIZE=512
TON_WORKER_INSERT_PARALLEL_ACTORS=3

and here's the logs i grep by command
cat docker.20240620-12h.log grep -v 'cell not found' | grep -v 'Rescheduling' | grep -v 'block not found'

ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 5][2024-06-19 17:55:14.064028833][IndexScheduler.cpp:172][!indexscheduler]        Last: 38469273 / 38469305       Blk/s: 0.253808 ETA: 2m 6s      Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 4][2024-06-19 17:55:24.075977842][IndexScheduler.cpp:172][!indexscheduler]        Last: 38469273 / 38469307       Blk/s: 0.088497 ETA: 6m 24s     Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 3][2024-06-19 17:55:34.088068994][IndexScheduler.cpp:172][!indexscheduler]        Last: 38469275 / 38469311       Blk/s: 0.116951 ETA: 5m 7s      Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 2][2024-06-19 17:55:44.100348436][IndexScheduler.cpp:172][!indexscheduler]        Last: 38469279 / 38469312       Blk/s: 0.268367 ETA: 2m 2s      Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 6][2024-06-19 17:55:54.110987234][IndexScheduler.cpp:172][!indexscheduler]        Last: 38469280 / 38469314       Blk/s: 0.174574 ETA: 3m 14s     Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 6][2024-06-19 17:56:04.124228634][IndexScheduler.cpp:172][!indexscheduler]        Last: 38469285 / 38469319       Blk/s: 0.440814 ETA: 1m 17s     Q[1M, 2b, 231t, 400m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 0][t 6][2024-06-19 17:56:07.133652766][ActorOwn.h:80][!slice]  Check `!empty()` failedESC[0m
ton-indexer-index-worker-1  | [pid 11] [time 1718819767] Signal: 6
ton-indexer-index-worker-1  | ------- Stack Backtrace -------
ton-indexer-index-worker-1  | ton-index-postgres(+0x35b162)[0x558c8f07f162]
ton-indexer-index-worker-1  | ton-index-postgres(+0x35b475)[0x558c8f07f475]
ton-indexer-index-worker-1  | ton-index-postgres(+0x35afbf)[0x558c8f07efbf]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f77644f2520]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f77645469fc]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f77644f2476]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f77644d87f3]
ton-indexer-index-worker-1  | ton-index-postgres(+0x101e44)[0x558c8ee25e44]
ton-indexer-index-worker-1  | ton-index-postgres(+0x380ef9)[0x558c8f0a4ef9]
ton-indexer-index-worker-1  | ton-index-postgres(+0x378996)[0x558c8f09c996]
ton-indexer-index-worker-1  | ton-index-postgres(+0x366aba)[0x558c8f08aaba]
ton-indexer-index-worker-1  | ton-index-postgres(+0x4b2e39)[0x558c8f1d6e39]
ton-indexer-index-worker-1  | ton-index-postgres(+0x4b355a)[0x558c8f1d755a]
ton-indexer-index-worker-1  | ton-index-postgres(+0x398bd6)[0x558c8f0bcbd6]
ton-indexer-index-worker-1  | ton-index-postgres(+0x398e1d)[0x558c8f0bce1d]
ton-indexer-index-worker-1  | ton-index-postgres(+0x3a99e3)[0x558c8f0cd9e3]
ton-indexer-index-worker-1  | ton-index-postgres(+0x39c7df)[0x558c8f0c07df]
ton-indexer-index-worker-1  | ton-index-postgres(+0x2a1b5a)[0x558c8efc5b5a]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f7764544ac3]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(clone+0x44)[0x7f77645d5a04]
ton-indexer-index-worker-1  | -------------------------------
ton-indexer-index-worker-1  | Postgres password file: /run/secrets/postgres_password
ton-indexer-index-worker-1  | Postgres host: postgres (ip: 192.168.64.2)
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 1][2024-06-19 17:56:09.900240451][InsertManagerBase.cpp:18][!insertmanager]       InsertManagerBase::start_up calledESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 1][2024-06-19 17:56:09.900333713][InsertManagerBase.cpp:29][!insertmanager]       Insert manager(parallel=12, max_batch_mc_blocks=2000, max_batch_blocks=2000, max_batch_txs=10000, max_batch_ms
gs=10000)ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 2][2024-06-19 17:56:09.901292155][InsertManagerPostgres.cpp:1139][!insertmanager] Reading existing seqnosESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 17:56:10.013268917][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38299753_0_4000000000000000_925603F00C0780DB90289DFB78E1A3431C2F270BAC46CBFF
F4BAEF1AAE710A58ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 17:56:10.013371332][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_0_e000000000000000_DA24F9363055A874BA0EA1B6472264A94840FB70921994A1
D86B7DE1EE72FF52ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 17:56:10.013384187][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_0_3000000000000000_4C0D11D42315B93C3640E14EF411DB1B7F5951F1FE70D21D
5822A73097DA5AB3ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 17:56:10.013395483][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_0_1000000000000000_BD49D3F87D87302E68D076605F7B20F3B1EBE54BF0C213BC
B535D7E2D968BED4ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 17:56:10.013412217][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38299753_-1_8000000000000000_2B66D44A24ECD5BB77E5E4615BA0E606F501B24DB1C303A
2D5D45A603392BF49ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 17:56:10.013423607][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_-1_8000000000000000_D0E749AE414202B2E84F12366BA9D08FD974546CBF5A54F
6469C9E04B86E128DESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 17:56:10.013433569][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_0_9000000000000000_9D9A652D40AFAD09DD75F4F887EBE415230BDC9CBAD8E929
5BE83C3BA55D8E89ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 17:56:10.013444152][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_0_b000000000000000_37FC5B27D643323304670205D967852E35231B291BDDF1C7
8702E8B457DA4D67ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 17:56:10.013453834][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_0_6000000000000000_199B4DE9A88F3741D972F2DB3FB8EE70E07CF8748BC49563
5DC91C27C999DE69ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 2][2024-06-19 17:56:10.035842339][IndexScheduler.cpp:80][!indexscheduler] Found 115285 existing seqnosESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 7][2024-06-19 17:56:11.036855540][IndexScheduler.cpp:172][!indexscheduler]        Last: 0 / 38354000      Blk/s: 0.000000 ETA: -8s        Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 7][2024-06-19 17:56:11.047042450][IndexScheduler.cpp:95][!indexscheduler] Skipped 115285 existing seqnosESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 7][2024-06-19 17:56:11.047115112][IndexScheduler.cpp:211][!indexscheduler]        Syncronization complete!ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 1][2024-06-19 17:56:21.041446861][IndexScheduler.cpp:172][!indexscheduler]        Last: 0 / 38469325      Blk/s: 0.000000 ETA: -8s        Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 3][2024-06-19 17:56:31.046156802][IndexScheduler.cpp:172][!indexscheduler]        Last: 38469288 / 38469329       Blk/s: 0.252000 ETA: 2m 42s     Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 6][2024-06-19 17:56:41.052334993][IndexScheduler.cpp:172][!indexscheduler]        Last: 38469289 / 38469332       Blk/s: 0.135697 ETA: 5m 16s     Q[0M, 0b, 0t, 0m]ESC[0m
@dnjef9duo
Copy link
Author

dnjef9duo commented Jun 20, 2024 •
edited
Loading

here's another case in same day.

ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 6][2024-06-19 19:18:07.311817206][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470413 / 38470568       Blk/s: 0.132504 ETA: 19m 29s    Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 7][2024-06-19 19:18:17.323430654][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470415 / 38470572       Blk/s: 0.184031 ETA: 14m 13s    Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 7][2024-06-19 19:18:27.335369615][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470416 / 38470573       Blk/s: 0.117312 ETA: 22m 18s    Q[1M, 3b, 342t, 591m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 2][2024-06-19 19:18:37.347217737][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470421 / 38470574       Blk/s: 0.396144 ETA: 6m 26s     Q[2M, 4b, 288t, 507m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 6][2024-06-19 19:18:47.359880756][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470421 / 38470576       Blk/s: 0.138127 ETA: 18m 42s    Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | [pid 11] [time 1718824735] Signal: 11
ton-indexer-index-worker-1  | ------- Stack Backtrace -------
ton-indexer-index-worker-1  | ton-index-postgres(+0x35b162)[0x55659bfde162]
ton-indexer-index-worker-1  | ton-index-postgres(+0x35b475)[0x55659bfde475]
ton-indexer-index-worker-1  | ton-index-postgres(+0x35afbf)[0x55659bfddfbf]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fc306d5f520]
ton-indexer-index-worker-1  | ton-index-postgres(+0x6fdd46)[0x55659c380d46]
ton-indexer-index-worker-1  | ton-index-postgres(+0x4dc180)[0x55659c15f180]
ton-indexer-index-worker-1  | ton-index-postgres(+0x4b3456)[0x55659c136456]
ton-indexer-index-worker-1  | ton-index-postgres(+0x398bd6)[0x55659c01bbd6]
ton-indexer-index-worker-1  | ton-index-postgres(+0x398e1d)[0x55659c01be1d]
ton-indexer-index-worker-1  | ton-index-postgres(+0x3a99e3)[0x55659c02c9e3]
ton-indexer-index-worker-1  | ton-index-postgres(+0x39c7df)[0x55659c01f7df]
ton-indexer-index-worker-1  | ton-index-postgres(+0x2a1b5a)[0x55659bf24b5a]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7fc306db1ac3]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(clone+0x44)[0x7fc306e42a04]
ton-indexer-index-worker-1  | -------------------------------
ton-indexer-index-worker-1  | Postgres password file: /run/secrets/postgres_password
ton-indexer-index-worker-1  | Postgres host: postgres (ip: 192.168.64.2)
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 1][2024-06-19 19:18:56.961869763][InsertManagerBase.cpp:18][!insertmanager]       InsertManagerBase::start_up calledESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 1][2024-06-19 19:18:56.961971428][InsertManagerBase.cpp:29][!insertmanager]       Insert manager(parallel=12, max_batch_mc_blocks=2000, max_batch_blocks=2000, max_batch_txs=10000, max_batch_ms
gs=10000)ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 2][2024-06-19 19:18:56.964037480][InsertManagerPostgres.cpp:1139][!insertmanager] Reading existing seqnosESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 19:18:57.025943299][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38299753_0_4000000000000000_925603F00C0780DB90289DFB78E1A3431C2F270BAC46CBFF
F4BAEF1AAE710A58ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 19:18:57.026020115][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_0_e000000000000000_DA24F9363055A874BA0EA1B6472264A94840FB70921994A1
D86B7DE1EE72FF52ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 19:18:57.026034345][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_0_3000000000000000_4C0D11D42315B93C3640E14EF411DB1B7F5951F1FE70D21D
5822A73097DA5AB3ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 19:18:57.026044682][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_0_1000000000000000_BD49D3F87D87302E68D076605F7B20F3B1EBE54BF0C213BC
B535D7E2D968BED4ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 19:18:57.026053299][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38299753_-1_8000000000000000_2B66D44A24ECD5BB77E5E4615BA0E606F501B24DB1C303A
2D5D45A603392BF49ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 19:18:57.026061551][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_-1_8000000000000000_D0E749AE414202B2E84F12366BA9D08FD974546CBF5A54F
6469C9E04B86E128DESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 19:18:57.026074456][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_0_9000000000000000_9D9A652D40AFAD09DD75F4F887EBE415230BDC9CBAD8E929
5BE83C3BA55D8E89ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 19:18:57.026087148][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_0_b000000000000000_37FC5B27D643323304670205D967852E35231B291BDDF1C7
8702E8B457DA4D67ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 19:18:57.026105759][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_0_6000000000000000_199B4DE9A88F3741D972F2DB3FB8EE70E07CF8748BC49563
5DC91C27C999DE69ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 2][2024-06-19 19:18:57.123912698][IndexScheduler.cpp:80][!indexscheduler] Found 116423 existing seqnosESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 2][2024-06-19 19:18:58.124357089][IndexScheduler.cpp:172][!indexscheduler]        Last: 0 / 38354000      Blk/s: 0.000000 ETA: -8s        Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 2][2024-06-19 19:18:58.136064095][IndexScheduler.cpp:95][!indexscheduler] Skipped 116423 existing seqnosESC[0m
ton-indexer-index-worker-1  | ESC[1;33m[ 2][t 3][2024-06-19 19:19:06.648128784][InterfaceDetectors.hpp:717][!nft_collection_detector]   Failed to parse collection content for 0:FD8C0CF95B84F3A6816D624E41840AA62C84CA85C57F9A290869E7FA12BC5
2AD: [Error : 0 : Unknown token data type]ESC[0m
ton-indexer-index-worker-1  | ESC[1;33m[ 2][t 3][2024-06-19 19:19:06.648193741][InterfaceDetectors.hpp:718][!nft_collection_detector]   te6cckEBAQEANwAAamlwZnM6Ly9RbWU2dU1MZHZYbVN1TUFQb3ZrY3E3bW83TFZvUFV3bmhrQ21MQXhhYThSRlVx3rPonA==ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 4][2024-06-19 19:19:08.130576114][IndexScheduler.cpp:172][!indexscheduler]        Last: 0 / 38470579      Blk/s: 0.000000 ETA: -8s        Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 2][2024-06-19 19:19:18.135292318][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470425 / 38470583       Blk/s: 0.077484 ETA: 33m 59s    Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 5][2024-06-19 19:19:28.140136987][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470425 / 38470586       Blk/s: 0.027017 ETA: 1h 39m 19s Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 5][2024-06-19 19:19:38.146084122][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470425 / 38470588       Blk/s: 0.009420 ETA: 4h 48m 23s Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 5][2024-06-19 19:19:48.150923460][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470425 / 38470591       Blk/s: 0.003285 ETA: 14h 2m 18s Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 6][2024-06-19 19:19:58.155169442][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470425 / 38470592       Blk/s: 0.001145 ETA: 1d 16h 30m 15s     Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 7][2024-06-19 19:20:08.160238788][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470425 / 38470595       Blk/s: 0.000399 ETA: 4d 22h 15m 7s      Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 1][2024-06-19 19:20:18.164655788][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470425 / 38470597       Blk/s: 0.000139 ETA: 14d 7h 7m 59s      Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 2][2024-06-19 19:20:28.170366672][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470425 / 38470598       Blk/s: 0.000049 ETA: 41d 5h 49m 4s      Q[0M, 0b, 0t, 0m]ESC[0m

@kdimentionaltree
Copy link
Collaborator

Hello, from your logs I see a successful restart of the worker. Could you please check if restart occurs due to Out Of Memory?

@dnjef9duo
Copy link
Author

dnjef9duo commented Jun 24, 2024 •
edited
Loading

i think there is no OOM.
there is no kind of message shows OOM or kill or something like that in system log and here is the monitor chart.
and we have 128g for ton node.

Screenshot 2024-06-24 at 3 48 56 PM

@dnjef9duo
Copy link
Author

unfortunately it happened again, quite similar logs.
can provide a full log (38MB, 6hours before docker restart) if needed.

8 times in 6 hours, after indexer auto restart, it syncing very low or even stop syncing.

2 cases of index-worker auto restart.
case1: signal 6

ton-indexer-index-worker-1  | ESC[1;31m[ 0][t 2][2024-06-25 19:54:54.409955543][ActorOwn.h:80][!slice]  Check `!empty()` failedESC[0m
ton-indexer-index-worker-1  | [pid 11] [time 1719345294] Signal: 6
ton-indexer-index-worker-1  | ------- Stack Backtrace -------
ton-indexer-index-worker-1  | ton-index-postgres(+0x35b162)[0x56398193e162]
ton-indexer-index-worker-1  | ton-index-postgres(+0x35b475)[0x56398193e475]
ton-indexer-index-worker-1  | ton-index-postgres(+0x35afbf)[0x56398193dfbf]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f114888e520]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f11488e29fc]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f114888e476]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f11488747f3]
ton-indexer-index-worker-1  | ton-index-postgres(+0x101e44)[0x5639816e4e44]
ton-indexer-index-worker-1  | ton-index-postgres(+0x380ef9)[0x563981963ef9]
ton-indexer-index-worker-1  | ton-index-postgres(+0x378996)[0x56398195b996]
ton-indexer-index-worker-1  | ton-index-postgres(+0x366aba)[0x563981949aba]
ton-indexer-index-worker-1  | ton-index-postgres(+0x4b2e39)[0x563981a95e39]
ton-indexer-index-worker-1  | ton-index-postgres(+0x4b355a)[0x563981a9655a]
ton-indexer-index-worker-1  | ton-index-postgres(+0x398bd6)[0x56398197bbd6]
ton-indexer-index-worker-1  | ton-index-postgres(+0x398e1d)[0x56398197be1d]
ton-indexer-index-worker-1  | ton-index-postgres(+0x3a99e3)[0x56398198c9e3]
ton-indexer-index-worker-1  | ton-index-postgres(+0x39c7df)[0x56398197f7df]
ton-indexer-index-worker-1  | ton-index-postgres(+0x2a1b5a)[0x563981884b5a]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f11488e0ac3]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(clone+0x44)[0x7f1148971a04]
ton-indexer-index-worker-1  | -------------------------------
ton-indexer-index-worker-1  | Postgres password file: /run/secrets/postgres_password
ton-indexer-index-worker-1  | Postgres host: postgres (ip: 192.168.96.2)
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 1][2024-06-25 19:54:55.781090741][InsertManagerBase.cpp:18][!insertmanager]       InsertManagerBase::start_up calledESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 1][2024-06-25 19:54:55.781210067][InsertManagerBase.cpp:29][!insertmanager]       Insert manager(parallel=12, max_batch_mc_blocks=2000, max_batch_blocks=2000, max_batch_txs=10000, max_batch_msgs=10000)ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 2][2024-06-25 19:54:55.782887243][InsertManagerPostgres.cpp:1139][!insertmanager] Reading existing seqnosESC[0m

case 2: signal 11

ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 3][2024-06-25 19:34:56.391172836][IndexScheduler.cpp:106][!indexquery]    Failed to fetch seqno 38599997: [Error : 651 : (0,2000000000000000,442240
85): ltdb: block not found]ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-25 19:34:56.391177588][IndexScheduler.cpp:106][!indexquery]    Failed to fetch seqno 38599998: [Error : 651 : (0,1000000000000000,442240
86): ltdb: block not found]ESC[0m
ton-indexer-index-worker-1  | ESC[1;33m[ 2][t 3][2024-06-25 19:34:56.391225825][IndexScheduler.cpp:117][!indexscheduler]        Rescheduling seqno 38599997ESC[0m
ton-indexer-index-worker-1  | ESC[1;33m[ 2][t 3][2024-06-25 19:34:56.391234462][IndexScheduler.cpp:117][!indexscheduler]        Rescheduling seqno 38599998ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 7][2024-06-25 19:34:56.391699328][IndexScheduler.cpp:106][!indexquery]    Failed to fetch seqno 38599995: [Error : 651 : (0,2000000000000000,442240
85): ltdb: block not found]ESC[0m
ton-indexer-index-worker-1  | ESC[1;33m[ 2][t 7][2024-06-25 19:34:56.391741690][IndexScheduler.cpp:117][!indexscheduler]        Rescheduling seqno 38599995ESC[0m
ton-indexer-index-worker-1  | [pid 11] [time 1719344096] Signal: 11
ton-indexer-index-worker-1  | ------- Stack Backtrace -------
ton-indexer-index-worker-1  | ton-index-postgres(+0x35b162)[0x560e5a75f162]
ton-indexer-index-worker-1  | ton-index-postgres(+0x35b475)[0x560e5a75f475]
ton-indexer-index-worker-1  | ton-index-postgres(+0x35afbf)[0x560e5a75efbf]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f14c0c4a520]
ton-indexer-index-worker-1  | ton-index-postgres(+0x6fdd46)[0x560e5ab01d46]
ton-indexer-index-worker-1  | ton-index-postgres(+0x4dc180)[0x560e5a8e0180]
ton-indexer-index-worker-1  | ton-index-postgres(+0x4b3456)[0x560e5a8b7456]
ton-indexer-index-worker-1  | ton-index-postgres(+0x398bd6)[0x560e5a79cbd6]
ton-indexer-index-worker-1  | ton-index-postgres(+0x398e1d)[0x560e5a79ce1d]
ton-indexer-index-worker-1  | ton-index-postgres(+0x3a99e3)[0x560e5a7ad9e3]
ton-indexer-index-worker-1  | ton-index-postgres(+0x39c7df)[0x560e5a7a07df]
ton-indexer-index-worker-1  | ton-index-postgres(+0x2a1b5a)[0x560e5a6a5b5a]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f14c0c9cac3]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(clone+0x44)[0x7f14c0d2da04]
ton-indexer-index-worker-1  | -------------------------------
ton-indexer-index-worker-1  | Postgres password file: /run/secrets/postgres_password
ton-indexer-index-worker-1  | Postgres host: postgres (ip: 192.168.96.2)
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 1][2024-06-25 19:34:58.491999860][InsertManagerBase.cpp:18][!insertmanager]       InsertManagerBase::start_up calledESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 1][2024-06-25 19:34:58.492225239][InsertManagerBase.cpp:29][!insertmanager]       Insert manager(parallel=12, max_batch_mc_blocks=2000, max_batch_b
locks=2000, max_batch_txs=10000, max_batch_msgs=10000)ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 1][2024-06-25 19:34:58.492252719][InsertManagerPostgres.cpp:1139][!insertmanager] Reading existing seqnosESC[0m

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kdimentionaltree @dnjef9duo