[Bug]: Invalid coroutine stack

[xiexiaoy]

(gdb) bt
#0  0x00007fa246a53304 in ontop_fcontext () from /mnt/ssd/jepsen-eloqdoc-test/eloqdoc-cloud/bin/../lib/libboost_context.so.1.74.0
Backtrace stopped: Cannot access memory at address 0x1002

logs.tar.zip

[xiexiaoy]

Another case

(gdb) bt
#0  0x00007fb68ac7b304 in ontop_fcontext () from /mnt/ssd/jepsen-eloqdoc-test/eloqdoc-cloud/bin/../lib/libboost_context.so.1.74.0
Backtrace stopped: Cannot access memory at address 0x1002
(gdb) p mongo::localThreadId
$1 = 3

INFO [2025-09-22 20:12:08,676] jepsen worker 1 - jepsen.mongodb.list-append skip client close
INFO [2025-09-22 20:12:08,677] jepsen worker 3 - jepsen.mongodb.list-append skip client close
INFO [2025-09-22 20:12:08,677] jepsen worker 5 - jepsen.mongodb.list-append skip client close
INFO [2025-09-22 20:12:08,691] jepsen worker 2 - jepsen.mongodb.list-append client closed
INFO [2025-09-22 20:12:08,693] jepsen worker 4 - jepsen.mongodb.list-append client closed
INFO [2025-09-22 20:12:08,727] jepsen worker 0 - jepsen.mongodb.list-append client closed
INFO [2025-09-22 20:12:08,762] real-pmap 1 - jepsen.mongodb.list-append skip client close
INFO [2025-09-22 20:12:08,765] real-pmap 0 - jepsen.mongodb.list-append client closed
INFO [2025-09-22 20:12:17,763] jepsen test runner - jepsen.core Run complete, writing
INFO [2025-09-22 20:12:23,972] jepsen test runner - jepsen.core Snarfing log files
INFO [2025-09-22 20:12:24,078] jepsen test runner - jepsen.core Snarfing log files
INFO [2025-09-22 20:12:24,193] jepsen node 192.168.122.23 - jepsen.mongodb.db Skip stop eloqdoc for test-external-eloqdoc is true.
INFO [2025-09-22 20:12:24,193] jepsen node 192.168.122.21 - jepsen.mongodb.db Skip stop eloqdoc for test-external-eloqdoc is true.
INFO [2025-09-22 20:12:24,196] jepsen test runner - jepsen.core Analyzing...
INFO [2025-09-22 20:12:33,435] jepsen test runner - jepsen.core Analysis complete
INFO [2025-09-22 20:12:33,452] jepsen results - jepsen.store Wrote /home/eloq/workspace/jepsen-eloqdoc/store/mongodb 7.0.0-rc8 list-append w:majority r:majority tw:majority tr:snapshot partition/20250922T200650.976+0800/results.edn
INFO [2025-09-22 20:12:33,481] jepsen test runner - jepsen.core {:perf {:latency-graph {:valid? true},
        :rate-graph {:valid? true},
        :valid? true},
 :clock {:valid? true},
 :stats {:valid? true,
         :count 256836,
         :ok-count 890,
         :fail-count 255926,
         :info-count 20,
         :by-f {:txn {:valid? true,
                      :count 256836,
                      :ok-count 890,
                      :fail-count 255926,
                      :info-count 20}}},
 :exceptions {:valid? true},
 :workload {:elle {:valid? true},
            :divergence {:valid? true,
                         :longest-count 188,
                         :diverged-count 0,
                         :diverged-frac 0.0},
            :valid? true},
 :crash {:valid? true, :count 0, :matches ()},
 :valid? true}


Everything looks good! <E3><83><BD>(<E2><80><98><E3><83><BC>`)<E3><83><8E>

$ cat run-jepsen.log                                                                                                                                                                                                              ✹ ✭eloqdoc-test 
~/workspace/jepsen-eloqdoc ~/workspace/jepsen-eloqdoc

=====[2025-09-22 20:06:40] test round 1/50 =====
drop db before next jepsen test
MongoDB shell version v4.0.3
connecting to: mongodb://192.168.122.23:27017/jepsendb
Implicit session: session { "id" : UUID("83c4c86e-a9f9-479b-9d3c-a4412cddce25") }
MongoDB server version: 4.0.3
{ "ok" : 1 }

running jepsen-test-with-partition with test-time(s) 300
[info] jepsen-test-with-partition completed !

drop db before next jepsen test
MongoDB shell version v4.0.3
connecting to: mongodb://192.168.122.23:27017/jepsendb
Implicit session: session { "id" : UUID("1f7cfb03-2773-42c8-8466-40d6c34a8bf7") }
MongoDB server version: 4.0.3
2025-09-22T20:14:03.867+0800 E QUERY    [js] Error: error doing query: failed: network error while attempting to run command 'dropDatabase' on host '192.168.122.23:27017'  :
DB.prototype.runCommand@src/mongo/shell/db.js:168:1
DB.prototype.dropDatabase@src/mongo/shell/db.js:428:16
@(shell eval):1:1
2025-09-22T20:14:03.873+0800 I NETWORK  [js] trying reconnect to 192.168.122.23:27017 failed
2025-09-22T20:14:04.029+0800 I NETWORK  [js] reconnect 192.168.122.23:27017 failed failed 
2025-09-22T20:14:04.029+0800 I QUERY    [js] Failed to end session { id: UUID("1f7cfb03-2773-42c8-8466-40d6c34a8bf7") } due to SocketException: socket exception [CONNECT_ERROR] server [network error while attempting to run command 'isMaster' on host '192.168.122.23:27017' ]

$ tail -n100000 eloqdoc.log  | grep -a '4295569248'
I20250922 20:12:35.541857 909944 tx_operation.cpp:2372] txn: 4295569248 process acquire_all_intent_op_
I20250922 20:12:35.548521 909944 tx_operation.cpp:2428] txn: 4295569248 process prepare_log_op_
I20250922 20:12:35.580099 909944 tx_operation.cpp:2508] txn: 4295569248 process post_all_intent_op_
I20250922 20:12:35.737088 909944 tx_operation.cpp:2547] txn: 4295569248 process acquire_all_lock_op_
W20250922 20:12:44.480312 909945 cc_shard.cpp:843] orphan lock detected, lock holding txn: 4295569248, txn is initiated by this machine, try to recover
I20250922 20:12:44.480336 909945 cc_shard.cpp:868] Txn #4295569248 locks __catalog, lock type: 3
W20250922 20:12:44.480350 909945 cc_shard.cpp:877] orphan lock detected, lock holding txn: 4295569248, try to recover
W20250922 20:12:44.481662 909944 cc_shard.cpp:843] orphan lock detected, lock holding txn: 4295569248, txn is initiated by this machine, try to recover
I20250922 20:12:44.481693 909944 cc_shard.cpp:868] Txn #4295569248 locks __catalog, lock type: 4
W20250922 20:12:44.481745 909944 cc_shard.cpp:877] orphan lock detected, lock holding txn: 4295569248, try to recover
I20250922 20:12:44.489426 909887 log_replay_service.cpp:126] replay service processes a RecoverTx task, txn: 4295569248
I20250922 20:12:44.490955 909887 log_replay_service.cpp:931] The tx 4295569248 is ongoing. Does nothing for recovery.
W20250922 20:12:44.509038 909943 cc_shard.cpp:843] orphan lock detected, lock holding txn: 4295569248, txn is initiated by this machine, try to recover
I20250922 20:12:44.509167 909943 cc_shard.cpp:868] Txn #4295569248 locks __catalog, lock type: 3
I20250922 20:12:44.509184 909943 cc_shard.cpp:868] Txn #4295569248 locks __cluster_config, lock type: 2
W20250922 20:12:44.509202 909943 cc_shard.cpp:877] orphan lock detected, lock holding txn: 4295569248, try to recover
W20250922 20:12:44.522708 909946 cc_shard.cpp:843] orphan lock detected, lock holding txn: 4295569248, txn is initiated by this machine, try to recover
I20250922 20:12:44.522728 909946 cc_shard.cpp:868] Txn #4295569248 locks __catalog, lock type: 3
W20250922 20:12:44.522744 909946 cc_shard.cpp:877] orphan lock detected, lock holding txn: 4295569248, try to recover
I20250922 20:12:44.896598 909887 log_replay_service.cpp:126] replay service processes a RecoverTx task, txn: 4295569248
I20250922 20:12:44.897387 909887 log_replay_service.cpp:931] The tx 4295569248 is ongoing. Does nothing for recovery.
I20250922 20:12:45.081653 909887 log_replay_service.cpp:126] replay service processes a RecoverTx task, txn: 4295569248
I20250922 20:12:45.083236 909887 log_replay_service.cpp:931] The tx 4295569248 is ongoing. Does nothing for recovery.
I20250922 20:12:45.459877 909887 log_replay_service.cpp:126] replay service processes a RecoverTx task, txn: 4295569248
I20250922 20:12:45.467968 909887 log_replay_service.cpp:931] The tx 4295569248 is ongoing. Does nothing for recovery.
W20250922 20:12:59.138427 909945 cc_shard.cpp:843] orphan lock detected, lock holding txn: 4295569248, txn is initiated by this machine, try to recover
I20250922 20:12:59.138447 909945 cc_shard.cpp:868] Txn #4295569248 locks __catalog, lock type: 3
W20250922 20:12:59.138459 909945 cc_shard.cpp:877] orphan lock detected, lock holding txn: 4295569248, try to recover
W20250922 20:12:59.142045 909944 cc_shard.cpp:843] orphan lock detected, lock holding txn: 4295569248, txn is initiated by this machine, try to recover
I20250922 20:12:59.142105 909944 cc_shard.cpp:868] Txn #4295569248 locks __catalog, lock type: 4
W20250922 20:12:59.142122 909944 cc_shard.cpp:877] orphan lock detected, lock holding txn: 4295569248, try to recover
I20250922 20:12:59.163228 909887 log_replay_service.cpp:126] replay service processes a RecoverTx task, txn: 4295569248
W20250922 20:12:59.173657 909946 cc_shard.cpp:843] orphan lock detected, lock holding txn: 4295569248, txn is initiated by this machine, try to recover
I20250922 20:12:59.173676 909946 cc_shard.cpp:868] Txn #4295569248 locks __catalog, lock type: 3
W20250922 20:12:59.173687 909946 cc_shard.cpp:877] orphan lock detected, lock holding txn: 4295569248, try to recover
W20250922 20:12:59.174074 909943 cc_shard.cpp:843] orphan lock detected, lock holding txn: 4295569248, txn is initiated by this machine, try to recover
I20250922 20:12:59.174225 909943 cc_shard.cpp:868] Txn #4295569248 locks __catalog, lock type: 3
I20250922 20:12:59.174270 909943 cc_shard.cpp:868] Txn #4295569248 locks __cluster_config, lock type: 2
W20250922 20:12:59.174289 909943 cc_shard.cpp:877] orphan lock detected, lock holding txn: 4295569248, try to recover
I20250922 20:12:59.180573 909887 log_replay_service.cpp:931] The tx 4295569248 is ongoing. Does nothing for recovery.
I20250922 20:12:59.446296 909887 log_replay_service.cpp:126] replay service processes a RecoverTx task, txn: 4295569248
I20250922 20:12:59.447144 909887 log_replay_service.cpp:931] The tx 4295569248 is ongoing. Does nothing for recovery.
I20250922 20:13:00.123504 909887 log_replay_service.cpp:126] replay service processes a RecoverTx task, txn: 4295569248
I20250922 20:13:00.176163 909887 log_replay_service.cpp:931] The tx 4295569248 is ongoing. Does nothing for recovery.
I20250922 20:13:00.232252 909887 log_replay_service.cpp:126] replay service processes a RecoverTx task, txn: 4295569248
I20250922 20:13:00.251174 909887 log_replay_service.cpp:931] The tx 4295569248 is ongoing. Does nothing for recovery.

log.tar.gz

[xiexiaoy]

PR #205 killAllExpiredTransactions clear coro status after executing abort.

[xiexiaoy]

logs.tar.zip

(gdb) bt
#0  0x00007f2985b7d304 in ontop_fcontext () from /lib/x86_64-linux-gnu/libboost_context.so.1.74.0
#1  0x0000000000000000 in ?? ()
(gdb) p mongo::localThreadid
No symbol "localThreadid" in namespace "mongo".
(gdb) p mongo::localThreadId
$1 = 1

[xiexiaoy]

killAllExpiredTransactions: use a shared_ptr to protect the source/stack.