doris1.2.1数据量上亿查询造成be挂掉，未查询时fe报出告警日志

[yangxd666]

fe日志：
2023-02-16 13:52:18,421 WARN (replayer|87) [Env.replayJournal():2522] replay journal cost too much time: 1002 replayedJournalId: 863481
2023-02-16 13:52:19,428 WARN (replayer|87) [BDBJournalCursor.next():147] Catch an exception when get next JournalEntity. key:863482
com.sleepycat.je.LockTimeoutException: (JE 18.3.12) Lock expired. Locker 696823970 -1_replayer_ReplicaThreadLocker: waited for lock on database=823672 LockAddr:23925506 LSN=0x2c/0x8d4430 type=READ grant=WAIT_NEW timeoutMillis=1000 startTime=1676526738427 endTime=1676526739427
Owners: [358595875 -1303619_ReplayThread_ReplayTxn" type="WRITE"/>]
Waiters: []

at com.sleepycat.je.txn.LockManager.makeTimeoutException(LockManager.java:1154) ~[je-18.3.13-doris-SNAPSHOT.jar:18.3.13-doris-SNAPSHOT]
at com.sleepycat.je.txn.LockManager.waitForLock(LockManager.java:621) ~[je-18.3.13-doris-SNAPSHOT.jar:18.3.13-doris-SNAPSHOT]
at com.sleepycat.je.txn.LockManager.lock(LockManager.java:353) ~[je-18.3.13-doris-SNAPSHOT.jar:18.3.13-doris-SNAPSHOT]
at com.sleepycat.je.txn.BasicLocker.lockInternal(BasicLocker.java:125) ~[je-18.3.13-doris-SNAPSHOT.jar:18.3.13-doris-SNAPSHOT]
at com.sleepycat.je.rep.txn.ReplicaThreadLocker.lockInternal(ReplicaThreadLocker.java:63) ~[je-18.3.13-doris-SNAPSHOT.jar:18.3.13-doris-SNAPSHOT]
at com.sleepycat.je.txn.Locker.lock(Locker.java:496) ~[je-18.3.13-doris-SNAPSHOT.jar:18.3.13-doris-SNAPSHOT]
at com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:3674) ~[je-18.3.13-doris-SNAPSHOT.jar:18.3.13-doris-SNAPSHOT]
at com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:3399) ~[je-18.3.13-doris-SNAPSHOT.jar:18.3.13-doris-SNAPSHOT]
at com.sleepycat.je.dbi.CursorImpl.lockLNAndCheckDefunct(CursorImpl.java:2177) ~[je-18.3.13-doris-SNAPSHOT.jar:18.3.13-doris-SNAPSHOT]
at com.sleepycat.je.dbi.CursorImpl.searchExact(CursorImpl.java:1989) ~[je-18.3.13-doris-SNAPSHOT.jar:18.3.13-doris-SNAPSHOT]
at com.sleepycat.je.Cursor.searchExact(Cursor.java:4210) ~[je-18.3.13-doris-SNAPSHOT.jar:18.3.13-doris-SNAPSHOT]
at com.sleepycat.je.Cursor.searchNoDups(Cursor.java:4071) ~[je-18.3.13-doris-SNAPSHOT.jar:18.3.13-doris-SNAPSHOT]
at com.sleepycat.je.Cursor.search(Cursor.java:3842) ~[je-18.3.13-doris-SNAPSHOT.jar:18.3.13-doris-SNAPSHOT]
at com.sleepycat.je.Cursor.getInternal(Cursor.java:1290) ~[je-18.3.13-doris-SNAPSHOT.jar:18.3.13-doris-SNAPSHOT]
at com.sleepycat.je.Database.get(Database.java:1403) ~[je-18.3.13-doris-SNAPSHOT.jar:18.3.13-doris-SNAPSHOT]
at com.sleepycat.je.Database.get(Database.java:1462) ~[je-18.3.13-doris-SNAPSHOT.jar:18.3.13-doris-SNAPSHOT]
at org.apache.doris.journal.bdbje.BDBJournalCursor.next(BDBJournalCursor.java:107) ~[doris-fe.jar:1.0-SNAPSHOT]
at org.apache.doris.catalog.Env.replayJournal(Env.java:2504) ~[doris-fe.jar:1.0-SNAPSHOT]
at org.apache.doris.catalog.Env$3.runOneCycle(Env.java:2292) ~[doris-fe.jar:1.0-SNAPSHOT]
at org.apache.doris.common.util.Daemon.run(Daemon.java:116) ~[doris-fe.jar:1.0-SNAPSHOT]

be亿级数据查询会直接挂掉部分节点 日志：
I0216 15:09:00.191443 24055 storage_engine.cpp:369] get root path info cost: 1 ms. tablet counter: 1222
I0216 15:09:00.192255 24055 task_worker_pool.cpp:1519] successfully report DISK|host=10.128.13.228|port=9020
I0216 15:09:01.965533 23485 daemon.cpp:214] physical memory 62.74 GB, process memory used 2.86 GB limit 50.20 GB, sys mem available 35.41 GB low water mark 1.60 GB
I0216 15:09:06.989869 23485 daemon.cpp:214] physical memory 62.74 GB, process memory used 2.86 GB limit 50.20 GB, sys mem available 35.41 GB low water mark 1.60 GB
I0216 15:09:09.654361 24054 task_worker_pool.cpp:1519] successfully report TASK|host=10.128.13.228|port=9020
I0216 15:09:10.533229 23974 olap_server.cpp:719] cooldown producer get tablet num: 0
I0216 15:09:12.016911 23485 daemon.cpp:214] physical memory 62.74 GB, process memory used 2.86 GB limit 50.20 GB, sys mem available 35.41 GB low water mark 1.60 GB
I0216 15:09:17.049346 23485 daemon.cpp:214] physical memory 62.74 GB, process memory used 2.86 GB limit 50.20 GB, sys mem available 35.41 GB low water mark 1.60 GB
I0216 15:09:21.820467 23849 load_channel_mgr.cpp:180] cleaning timed out load channels
I0216 15:09:21.820513 23849 load_channel_mgr.cpp:212] load mem consumption(bytes). limit: 26948281958, current: 0, peak: 0, total running load channels: 0
I0216 15:09:22.075503 23485 daemon.cpp:214] physical memory 62.74 GB, process memory used 2.86 GB limit 50.20 GB, sys mem available 35.41 GB low water mark 1.60 GB
I0216 15:09:22.340608 41184 heartbeat_server.cpp:47] get heartbeat from FE.host:10.128.13.228, port:9020, cluster id:1689904769, counter:290569
I0216 15:09:22.655321 24054 task_worker_pool.cpp:1519] successfully report TASK|host=10.128.13.228|port=9020
I0216 15:09:24.905200 23934 storage_engine.cpp:634] start trash and snapshot sweep.
I0216 15:09:24.906502 23934 storage_engine.cpp:369] get root path info cost: 1 ms. tablet counter: 1222
I0216 15:09:24.906579 23934 storage_engine.cpp:659] Start to sweep path /data/apache-doris/doris1
I0216 15:09:24.906821 23934 storage_engine.cpp:659] Start to sweep path /data/apache-doris/doris2
I0216 15:09:25.907840 23934 storage_engine.cpp:767] remove 0 invalid rowset meta from dir: /data/apache-doris/doris1
I0216 15:09:25.908030 23934 storage_engine.cpp:767] remove 0 invalid rowset meta from dir: /data/apache-doris/doris2
I0216 15:09:27.100819 23485 daemon.cpp:214] physical memory 62.74 GB, process memory used 2.86 GB limit 50.20 GB, sys mem available 35.35 GB low water mark 1.60 GB
I0216 15:09:30.533816 23974 olap_server.cpp:719] cooldown producer get tablet num: 0
I0216 15:09:32.124446 23485 daemon.cpp:214] physical memory 62.74 GB, process memory used 2.86 GB limit 50.20 GB, sys mem available 35.41 GB low water mark 1.60 GB
I0216 15:09:37.148279 23485 daemon.cpp:214] physical memory 62.74 GB, process memory used 2.86 GB limit 50.20 GB, sys mem available 35.41 GB low water mark 1.60 GB
I0216 15:09:37.656215 24054 task_worker_pool.cpp:1519] successfully report TASK|host=10.128.13.228|port=9020
I0216 15:09:42.175437 23485 daemon.cpp:214] physical memory 62.74 GB, process memory used 2.86 GB limit 50.20 GB, sys mem available 35.41 GB low water mark 1.60 GB
I0216 15:09:47.197374 23485 daemon.cpp:214] physical memory 62.74 GB, process memory used 2.86 GB limit 50.20 GB, sys mem available 35.41 GB low water mark 1.60 GB
I0216 15:09:49.657205 24054 task_worker_pool.cpp:1519] successfully report TASK|host=10.128.13.228|port=9020
I0216 15:09:50.534384 23974 olap_server.cpp:719] cooldown producer get tablet num: 0
I0216 15:09:51.959069 24056 tablet_manager.cpp:868] find expired transactions for 0 tablets
I0216 15:09:51.961277 24056 tablet_manager.cpp:906] success to build all report tablets info. tablet_count=1222
I0216 15:09:51.964538 24056 task_worker_pool.cpp:1519] successfully report TABLET|host=10.128.13.228|port=9020
I0216 15:09:52.220435 23485 daemon.cpp:214] physical memory 62.74 GB, process memory used 2.86 GB limit 50.20 GB, sys mem available 35.41 GB low water mark 1.60 GB
I0216 15:09:57.247591 23485 daemon.cpp:214] physical memory 62.74 GB, process memory used 2.86 GB limit 50.20 GB, sys mem available 35.41 GB low water mark 1.60 GB
I0216 15:10:02.271385 23485 daemon.cpp:214] physical memory 62.74 GB, process memory used 2.86 GB limit 50.20 GB, sys mem available 35.41 GB low water mark 1.60 GB
I0216 15:10:03.657263 24054 task_worker_pool.cpp:1519] successfully report TASK|host=10.128.13.228|port=9020
I0216 15:10:03.997696 24146 fragment_mgr.cpp:629] query_id: 9d3e7127f0e74ed9-bdfe1f49e4464618 coord_addr TNetworkAddress(hostname=10.128.13.219, port=9020) total fragment num on current host: 1
I0216 15:10:03.997802 24146 fragment_mgr.cpp:679] Register query/load memory tracker, query/load id: 9d3e7127f0e74ed9-bdfe1f49e4464618 limit: 2.00 GB
I0216 15:10:03.997850 24146 plan_fragment_executor.cpp:87] PlanFragmentExecutor::prepare|query_id=9d3e7127f0e74ed9-bdfe1f49e4464618|instance_id=9d3e7127f0e74ed9-bdfe1f49e4464619|backend_num=1|pthread_id=140510524475136
I0216 15:10:04.025148 23709 fragment_mgr.cpp:493] PlanFragmentExecutor::_exec_actual|query_id=9d3e7127f0e74ed9-bdfe1f49e4464618|instance_id=9d3e7127f0e74ed9-bdfe1f49e4464619|pthread_id=140513837442816
I0216 15:10:04.025200 23709 plan_fragment_executor.cpp:232] PlanFragmentExecutor::open|query_id=9d3e7127f0e74ed9-bdfe1f49e4464618|instance_id=9d3e7127f0e74ed9-bdfe1f49e4464619|mem_limit=2.00 GB
I0216 15:10:04.191630 24055 data_dir.cpp:734] path: /data/apache-doris/doris1 total capacity: 1082120392704, available capacity: 783974354944
I0216 15:10:04.191663 24055 data_dir.cpp:734] path: /data/apache-doris/doris2 total capacity: 1082120392704, available capacity: 783974354944
I0216 15:10:04.192684 24055 storage_engine.cpp:369] get root path info cost: 1 ms. tablet counter: 1222
I0216 15:10:04.193385 24055 task_worker_pool.cpp:1519] successfully report DISK|host=10.128.13.228|port=9020
I0216 15:10:05.625542 23709 plan_fragment_executor.cpp:699] Close() fragment_instance_id=9d3e7127f0e74ed9-bdfe1f49e4464619
I0216 15:10:05.625658 23709 query_fragments_ctx.h:55] Deregister query/load memory tracker, queryId=9d3e7127f0e74ed9-bdfe1f49e4464618, Limit=2.00 GB, CurrUsed=135.50 MB, PeakUsed=583.32 MB
I0216 15:10:07.294201 23485 daemon.cpp:214] physical memory 62.74 GB, process memory used 3.14 GB limit 50.20 GB, sys mem available 35.13 GB low water mark 1.60 GB
I0216 15:10:08.913022 53563 backend_service.cpp:351] get_batch stream_load_record rocksdb successfully. records size: 0, last_stream_load_timestamp: -1
I0216 15:10:10.535105 23974 olap_server.cpp:719] cooldown producer get tablet num: 0
I0216 15:10:12.317459 23485 daemon.cpp:214] physical memory 62.74 GB, process memory used 3.14 GB limit 50.20 GB, sys mem available 34.86 GB low water mark 1.60 GB
I0216 15:10:17.340373 23485 daemon.cpp:214] physical memory 62.74 GB, process memory used 3.14 GB limit 50.20 GB, sys mem available 34.71 GB low water mark 1.60 GB
I0216 15:10:17.658115 24054 task_worker_pool.cpp:1519] successfully report TASK|host=10.128.13.228|port=9020
I0216 15:10:19.397096 24152 internal_service.cpp:393] cancel fragment, fragment_instance_id=9d3e7127f0e74ed9-bdfe1f49e4464619, reason: 3
I0216 15:10:19.483778 24156 socket.cpp:2202] Checking Socket{id=514 addr=10.128.13.219:8060} (0x7fcc2f7d2d00)
I0216 15:10:21.331496 24143 socket.cpp:2202] Checking Socket{id=1025 addr=10.128.13.220:8060} (0x7fcae6a9c080)
I0216 15:10:21.820652 23849 load_channel_mgr.cpp:180] cleaning timed out load channels
I0216 15:10:21.820689 23849 load_channel_mgr.cpp:212] load mem consumption(bytes). limit: 26948281958, current: 0, peak: 0, total running load channels: 0
I0216 15:10:22.362792 23485 daemon.cpp:214] physical memory 62.74 GB, process memory used 3.14 GB limit 50.20 GB, sys mem available 35.13 GB low water mark 1.60 GB
I0216 15:10:22.578099 41184 heartbeat_server.cpp:47] get heartbeat from FE.host:10.128.13.228, port:9020, cluster id:1689904769, counter:290581
I0216 15:10:27.387861 23485 daemon.cpp:214] physical memory 62.74 GB, process memory used 3.14 GB limit 50.20 GB, sys mem available 35.13 GB low water mark 1.60 GB
I0216 15:10:30.535780 23974 olap_server.cpp:719] cooldown producer get tablet num: 0
I0216 15:10:31.659086 24054 task_worker_pool.cpp:1519] successfully report TASK|host=10.128.13.228|port=9020
I0216 15:10:32.411208 23485 daemon.cpp:214] physical memory 62.74 GB, process memory used 3.14 GB limit 50.20 GB, sys mem available 35.13 GB low water mark 1.60 GB
I0216 15:10:37.444980 23485 daemon.cpp:214] physical memory 62.74 GB, process memory used 3.14 GB limit 50.20 GB, sys mem available 35.13 GB low water mark 1.60 GB
I0216 15:10:42.467949 23485 daemon.cpp:214] physical memory 62.74 GB, process memory used 3.14 GB limit 50.20 GB, sys mem available 35.13 GB low water mark 1.60 GB
I0216 15:10:42.659983 24054 task_worker_pool.cpp:1519] successfully report TASK|host=10.128.13.228|port=9020
I0216 15:10:47.495323 23485 daemon.cpp:214] physical memory 62.74 GB, process memory used 3.14 GB limit 50.20 GB, sys mem available 35.13 GB low water mark 1.60 GB
I0216 15:10:50.536586 23974 olap_server.cpp:719] cooldown producer get tablet num: 0
I0216 15:10:52.519826 23485 daemon.cpp:214] physical memory 62.74 GB, process memory used 3.14 GB limit 50.20 GB, sys mem available 35.13 GB low water mark 1.60 GB
I0216 15:10:52.965273 24056 tablet_manager.cpp:868] find expired transactions for 0 tablets
I0216 15:10:52.967545 24056 tablet_manager.cpp:906] success to build all report tablets info. tablet_count=1222
I0216 15:10:52.971232 24056 task_worker_pool.cpp:1519] successfully report TABLET|host=10.128.13.228|port=9020
I0216 15:10:55.661034 24054 task_worker_pool.cpp:1519] successfully report TASK|host=10.128.13.228|port=9020
I0216 15:10:57.546924 23485 daemon.cpp:214] physical memory 62.74 GB, process memory used 3.14 GB limit 50.20 GB, sys mem available 35.13 GB low water mark 1.60 GB