Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Node suddenly force-closed 40 channels - Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel #2847

Closed
pavlenex opened this issue Jul 25, 2019 · 5 comments · Fixed by #2842

Comments

@pavlenex
Copy link

commented Jul 25, 2019

Issue and Steps to Reproduce

I've noticed that the majority of the channels are now closed, pending or offline. For now, funds have not been retrieved, but probably because of 144 limbo.

Logs

The chain is fully synched
Waiting /var/lib/tor/hidden_services/c-lightning/hostname to be created by tor...
announce-addr=fj2b35ny3kaxessk6su3ydbeh3ppqmxykqzbshesf43kzwgmmrjp2sid.onion:9735 added to /root/.lightning/config
C-Lightning starting, listening on port 9735
2019-07-25T15:01:35.198Z lightningd(6): --------------------------------------------------
2019-07-25T15:01:35.198Z lightningd(6): Server started with public key 02d35ad6428b083d63575e3b0ce33e06c507910c131970b07dc194953b43a9794c, alias BitcoinShirt (color #02d35a) and lightningd basedon-v0.7.1
2019-07-25T15:02:23.094Z lightningd(6): 03d37fca0656558de4fd86bbe490a38d84a46228e7ec1361801f54f9437a18d618 chan #9225: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel 47ac844a9b91f4365a4334164a3bf4f1cf688a24d1d0ff049bfecfcdb389ecc8: sync error
2019-07-25T15:02:23.094Z lightningd(6): 	 (tx edeaec247dbba4e1399b07130349cede852c40dfecb012940b9da329dd330765)
2019-07-25T15:02:23.148Z lightningd(6): 03d37fca0656558de4fd86bbe490a38d84a46228e7ec1361801f54f9437a18d618 chan #9225: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
2019-07-25T15:02:23.643Z lightningd(6): 030995c0c0217d763c2274aa6ed69a0bb85fa2f7d118f93631550f3b6219a577f5 chan #4513: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel 8bbf90cb5993fc3fc58949513a86664fe7ea90897e7d7fed9ff3de84d7540914: sync error
2019-07-25T15:02:23.643Z lightningd(6): 	 (tx e6ecbba710a177e40d01e136df347afe2e8da6ce44f307b1e596991ec647ed1b)
2019-07-25T15:02:23.868Z lightningd(6): 030995c0c0217d763c2274aa6ed69a0bb85fa2f7d118f93631550f3b6219a577f5 chan #4513: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
2019-07-25T15:02:23.974Z lightningd(6): 024d2387409269f3b79e2708bb39b895c9f4b6a8322153af54eba487d4993bf60f chan #4517: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel 5c4d358ea424710b18bd1f80921a2efad56e2d17e010cc2cdb48e2281d9cbe6d: sync error
2019-07-25T15:02:23.974Z lightningd(6): 	 (tx fd39b5e41a51d8d322023f1ab1e0050e1db3f24c8aa528a1caa968d4a58d606f)
2019-07-25T15:02:24.137Z lightningd(6): 024d2387409269f3b79e2708bb39b895c9f4b6a8322153af54eba487d4993bf60f chan #4517: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
2019-07-25T15:02:24.295Z lightningd(6): 0303a518845db99994783f606e6629e705cfaf072e5ce9a4d8bf9e249de4fbd019 chan #4514: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel fc59e0de7901f34d8a971885c253c94f48a9d5fdc4b843a0fb160dded0076290: sync error
2019-07-25T15:02:24.295Z lightningd(6): 	 (tx 515c8b19d0291d44eb15be078e7d15380932801543668cf4799d806a8244d1af)
2019-07-25T15:02:24.481Z lightningd(6): 0303a518845db99994783f606e6629e705cfaf072e5ce9a4d8bf9e249de4fbd019 chan #4514: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
2019-07-25T15:02:24.621Z lightningd(6): 03dab87ff8635982815c4567eb58af48f9944d11c56beb12b91e1049aaea06e187 chan #7015: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel 99444d205d3334c7d3bc0dd0b748b925c56b5b2636ede919f3c1df9f9f05b52a: sync error
2019-07-25T15:02:24.621Z lightningd(6): 	 (tx 3c86707e516de4badb34690f3316e4e451e9aa1e2d52afc0063adb88c9521a79)
2019-07-25T15:02:24.689Z lightningd(6): 03dab87ff8635982815c4567eb58af48f9944d11c56beb12b91e1049aaea06e187 chan #7015: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
2019-07-25T15:02:24.829Z lightningd(6): 03fce165537aea120bffe8505876b44d5119354f825b3eac329b761fc5636bf334 chan #4515: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel 380e535f2da60c71cdbacf0351bc6c12e370961e202f1f1d51be279abff14a3f: sync error
2019-07-25T15:02:24.829Z lightningd(6): 	 (tx e3f7c5eabb1623ab13ecf3ab1154fbdd0df98fcfd066fc2cf5cb39601c4b6c49)
2019-07-25T15:02:24.830Z lightningd(6): 03fce165537aea120bffe8505876b44d5119354f825b3eac329b761fc5636bf334 chan #4515: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
2019-07-25T15:02:25.048Z lightningd(6): 03da1c27ca77872ac5b3e568af30673e599a47a5e4497f85c7b5da42048807b3ed chan #7022: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel 713c020973a1f4b514734cf0d1beac160ffe90734a01f2087463ed9c9f8c7bff: sync error
2019-07-25T15:02:25.048Z lightningd(6): 	 (tx 193be06dcd3a75f26545026cb13769d4bb65de0e396dd672e2cd2df413bc39b6)
2019-07-25T15:02:25.255Z lightningd(6): 03da1c27ca77872ac5b3e568af30673e599a47a5e4497f85c7b5da42048807b3ed chan #7022: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
2019-07-25T15:02:25.357Z lightningd(6): 03bc9337c7a28bb784d67742ebedd30a93bacdf7e4ca16436ef3798000242b2251 chan #9702: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel c6fb1a0604488d5224dbd9ecc35b2e39088c7e7812f715fb26ec854cf42bbfa1: sync error
2019-07-25T15:02:25.357Z lightningd(6): 	 (tx 6880208215eaaf23fad2bc219649d60c75c189fad9e6c6b005fb8ad3e30cdf65)
2019-07-25T15:02:25.368Z lightningd(6): 03bc9337c7a28bb784d67742ebedd30a93bacdf7e4ca16436ef3798000242b2251 chan #9702: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
2019-07-25T15:02:25.695Z lightningd(6): 022c260f9ad58196af280c80a96ec9eabf6404df59ff1a7553b0f381c875a29ba0 chan #2419: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel 887789be8d181d1def71c078ffa09a74e44550e7950e53729a7a05aeb98a0398: sync error
2019-07-25T15:02:25.695Z lightningd(6): 	 (tx 8f6f886c12520203b43279a26d7faca3b4c2d8320b6fefca8dec2f0f9526c0e7)
2019-07-25T15:02:25.805Z lightningd(6): 022c260f9ad58196af280c80a96ec9eabf6404df59ff1a7553b0f381c875a29ba0 chan #2419: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
2019-07-25T15:02:25.876Z lightningd(6): 02247d9db0dfafea745ef8c9e161eb322f73ac3f8858d8730b6fd97254747ce76b chan #9681: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel d887a22c0424ce3f931f398810f81df34a008994406d11580aba081d1ff165d6: sync error
2019-07-25T15:02:25.876Z lightningd(6): 	 (tx 826052b9dc8570c5d3bbc8d20b489f5bb62075ab1a84767a3cc9a67ddf745893)
2019-07-25T15:02:25.881Z lightningd(6): 02247d9db0dfafea745ef8c9e161eb322f73ac3f8858d8730b6fd97254747ce76b chan #9681: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
2019-07-25T15:02:25.927Z lightningd(6): 034ea80f8b148c750463546bd999bf7321a0e6dfc60aaf84bd0400a2e8d376c0d5 chan #7571: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel eace7ef5e7afa2d2a0ee6bf1798c9b3625eb0f5d4cef2a11351cac60f50f21b1: sync error
2019-07-25T15:02:25.927Z lightningd(6): 	 (tx f601adb189d056ceb16f062f24e9cd3d4b4ee62f887f010aba945cd25e02b14d)
2019-07-25T15:02:25.973Z lightningd(6): 034ea80f8b148c750463546bd999bf7321a0e6dfc60aaf84bd0400a2e8d376c0d5 chan #7571: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
2019-07-25T15:02:26.037Z lightningd(6): 02c91d6aa51aa940608b497b6beebcb1aec05be3c47704b682b3889424679ca490 chan #7020: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel 4eaf9d038e97a5591f7877d472a43edc7be0b8bb9dbd207508917c09dbcad0b4: sync error
2019-07-25T15:02:26.037Z lightningd(6): 	 (tx f5f8485c98d02982c6381846d22e3655e66dd5c6f671c9d1091d3bad250e05ce)
2019-07-25T15:02:26.054Z lightningd(6): 02c91d6aa51aa940608b497b6beebcb1aec05be3c47704b682b3889424679ca490 chan #7020: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
2019-07-25T15:02:26.121Z lightningd(6): 02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b chan #7457: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel 6b8aa886c4b92d401fbf5bee99fc1e4bb7602577fff98b7e3009deeac6bf0a18: sync error
2019-07-25T15:02:26.121Z lightningd(6): 	 (tx d43d014ad5754b191bf1bdaed405fa32223e6a93b0791a652da15027eddbf354)
2019-07-25T15:02:26.123Z lightningd(6): 02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b chan #7457: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
2019-07-25T15:02:26.191Z lightningd(6): 0390b5d4492dc2f5318e5233ab2cebf6d48914881a33ef6a9c6bcdbb433ad986d0 chan #4512: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel f9cc157e6475088a2ce749063993e9a537d98758a440ada49d7ac68222fcbfd7: sync error
2019-07-25T15:02:26.191Z lightningd(6): 	 (tx f2d74cfa26cb99a513ba9bebb0f3379fd9677366ea70d6c125360771e482c908)
2019-07-25T15:02:26.237Z lightningd(6): 0390b5d4492dc2f5318e5233ab2cebf6d48914881a33ef6a9c6bcdbb433ad986d0 chan #4512: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
2019-07-25T15:02:26.282Z lightningd(6): 02725e5abcbf5550fc29e6b19706a1377f25d2b1502684f9be9965b6deac167520 chan #8461: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel 3c9bc542d17c02abaee1d56e1a902f190d690c24af42cc8a9fae6cb768e565e6: sync error
2019-07-25T15:02:26.282Z lightningd(6): 	 (tx 36a5a59342a5c4bc9684fa75447835fb84c04348869a3fc1bdec2bc1ac049dad)
2019-07-25T15:02:26.365Z lightningd(6): 02725e5abcbf5550fc29e6b19706a1377f25d2b1502684f9be9965b6deac167520 chan #8461: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
2019-07-25T15:02:31.894Z lightningd(6): 02e708bcd8685044fec60699809f5e1d5523833e2c1aec3cf96dd1dedf0ba0db78 chan #17229: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel d5b5fa3462e7a1d899cf4a79a479fb84e00e589ef76500b27f0c7fd7aa96dbdb: sync error
2019-07-25T15:02:31.894Z lightningd(6): 	 (tx ffc3486aa7c568e1ee1ea22ac754dd411f823cf10966c0b3b82a15819f5fd9f5)
2019-07-25T15:02:31.902Z lightningd(6): 02e708bcd8685044fec60699809f5e1d5523833e2c1aec3cf96dd1dedf0ba0db78 chan #17229: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
2019-07-25T15:02:32.213Z lightningd(6): 022ec48186f2366b429d2fca1af2f4fc33dd97a8f42c55ab4c4ab4947f8a3cf768 chan #17833: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel 388f356343c8ccb0bcbabcb96c7e7a4bfb6d4b245eeaab5f0119d0b3b252ae6d: sync error
2019-07-25T15:02:32.213Z lightningd(6): 	 (tx 703837e5110a282a5e4b6b8253e42d3f1f116afb9b21b6511a619b688148830d)
2019-07-25T15:02:32.214Z lightningd(6): 022ec48186f2366b429d2fca1af2f4fc33dd97a8f42c55ab4c4ab4947f8a3cf768 chan #17833: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
2019-07-25T15:02:32.242Z lightningd(6): 02309f3556d51e9d855594762dcf68cb283fbfd7e48618b6cecb8cd90e2e34d8ad chan #2216: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel 03ec803d8cef5fe03f78cdd49a19e01973af69f69c480aa9d232aa4f9ffa652c: sync error
2019-07-25T15:02:32.242Z lightningd(6): 	 (tx 3c32cd613bb2078c286c6e10860417c0c9ac796d6663763209ce3686640567a7)
2019-07-25T15:02:32.281Z lightningd(6): 02309f3556d51e9d855594762dcf68cb283fbfd7e48618b6cecb8cd90e2e34d8ad chan #2216: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
2019-07-25T15:02:32.623Z lightningd(6): 03323b53ff21a3d5d0b87c4a94e85bc65a136a93f54d255cb9069afac3b9a40169 chan #9929: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel c30de988fd8aa37e35717724e250011ba301434862c76963bb5a1a2f662d2503: sync error
2019-07-25T15:02:32.623Z lightningd(6): 	 (tx 4a367dc136e476cec7401ae393bd5183f4eec1551df3ab9c172a319889ef76b4)
2019-07-25T15:02:32.632Z lightningd(6): 03323b53ff21a3d5d0b87c4a94e85bc65a136a93f54d255cb9069afac3b9a40169 chan #9929: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
2019-07-25T15:02:32.743Z lightningd(6): 03a08d40fc3676e62a0bcaa27de657ce5e05b9a2491fbbf011a506a774e9ea263a chan #12073: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel 7d8619116aeaa4dbc78562f2468abca4902df17163bce343c6f209d57909c0b1: sync error
2019-07-25T15:02:32.743Z lightningd(6): 	 (tx daca27a09aa0d3b8d0256195a8ad744c9e5e1e5ddd4073b29d74040610be2a94)
2019-07-25T15:02:32.800Z lightningd(6): 03a08d40fc3676e62a0bcaa27de657ce5e05b9a2491fbbf011a506a774e9ea263a chan #12073: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
2019-07-25T15:02:33.094Z lightningd(6): 029d3b9c46beb7303c80ab700f51652f54f472e92ffdcb6c618e8247b19827061a chan #17481: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel d5fa53f03bdb493ba0a7c80db03944daa0b61b1e1504887140ef3526fb5b9d26: sync error
2019-07-25T15:02:33.094Z lightningd(6): 	 (tx 4797a74a87f64d01b04ea82494477612ea44d947467d8d5f466f27a63e466aa8)
2019-07-25T15:02:33.101Z lightningd(6): 029d3b9c46beb7303c80ab700f51652f54f472e92ffdcb6c618e8247b19827061a chan #17481: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
2019-07-25T15:02:36.201Z lightningd(6): bitcoin-cli: finished bitcoin-cli -datadir=/etc/bitcoin -rpcconnect=bitcoind sendrawtransaction 0200000000010147ac844a9b91f4365a4334164a3bf4f1cf688a24d1d0ff049bfecfcdb389ecc80000000000173655800235a10000000000002200209e9bc01d0640be5d435fae5820473fc9f5568d2ffcc7da4701edefd4ca9348081d473c0000000000160014b92404cc9c21000539690865c2b01c3fb1c10dd504004730440220607a4ed68e338f4cb932317e7176ad7eb782f29f638eef7b2a51eacd9c7f7b520220253da2d385409774408e9cebaf2e326061f6dd872f0f4d90b8db3a725157df2e01483045022100b43141b3832e657040022f18692fb86f71030ed41874a5038ddff4d87629eddd022050f326488fcfb7ce54d2502129c2a08a2a335ce1592fcc5f80856f6da92beda701475221031741911b3dc64a81ab81a1177d0e8c6f306651fbdaf651c38e20c6a9e737f7dd2103f8553d4d6d2f0fdd58be56ab8c9933097d4040c328d02a222008951f9a882e6252ae75889620 (13052 ms)
2019-07-25T15:02:36.233Z lightningd(6): bitcoin-cli: finished bitcoin-cli -datadir=/etc/bitcoin -rpcconnect=bitcoind sendrawtransaction 020000000001018bbf90cb5993fc3fc58949513a86664fe7ea90897e7d7fed9ff3de84d75409150100000000f0b9d280023c780100000000002200202dbb2c11b44f439d1b9fbcee147fceaa237bdcd2efcb1074aea3a3f9d409b23e54bb870000000000160014b1d4918583db3377b97e8da0695c5e99db951b070400483045022100908f9ad4b020785b80ea0e4725843f79019ab7b4520939f13f2b525a1dc02c4802205b458004d620f0b517807ec65c1b9c5d60a762ee1ff1245b022250cdde62966c0147304402204050a42870f07de1c123a52601bd068257d9e8749c21a626d1f67ae04f68149c022072740a19699cfe4ff50d4aec7777abe9d05f1451cf1c3fd5759049dab4ceac9f0147522102af6ca560b50f5183fe78fc191345fb3fec93bfe6220a2d1e3590b95cab11a6152102dcc3d9dc3ad6c5e41c8ba6515d32bd854b483b6fb2fdbbccd3333be4a390506f52aea2622d20 (12365 ms)
2019-07-25T15:02:36.251Z lightningd(6): bitcoin-cli: finished bitcoin-cli -datadir=/etc/bitcoin -rpcconnect=bitcoind sendrawtransaction 020000000001015c4d358ea424710b18bd1f80921a2efad56e2d17e010cc2cdb48e2281d9cbe6c0100000000d14e8280028265010000000000220020690173fe6c7334aa913ffd360afd234bb54c0aa3589ab4f08435cc1d6fe0c71b18ce870000000000160014119a731c86ffe1fe75341a3a8b00bd6f4bd961500400473044022055eaf4e370cfde0ab056cc229698b26993e4961f3d964837027e578e16786a2c02205ec43c352fe4f172f22d7747247efbcd8fdb59e908f876469f8de511fcebd56901483045022100f26d22c5b75da4bbd9122837c8e31f9018c0e120301b9d1404609f91527c3c0802205feb7daebd7eeb610555482958d4d795cda6fe1e5e4e6ee16d0955883d5d0c430147522102dcdd2b1cee574bb08ec593d2f3014e9fd83133b0c000c2b56a50fec1f60d170221038b4defb97dd877764b986d3afcce505acce3ae7e1b559a69bb14672978b9ac4052ae1d70c320 (12114 ms)
2019-07-25T15:02:36.285Z lightningd(6): bitcoin-cli: finished bitcoin-cli -datadir=/etc/bitcoin -rpcconnect=bitcoind sendrawtransaction 02000000000101fc59e0de7901f34d8a971885c253c94f48a9d5fdc4b843a0fb160dded00762900000000000e327588002188c0100000000002200204654a5e8114722a58d772e549589b7f0f422cea600be97cb6707dd6a2c5e19bd7aa787000000000016001463afa047c3085859f9a15615ef92813ac1e3b7e4040047304402203617f02123c912cd03ec7e6a2856df27941f82e9e4b1395137ad14b6c0767bb90220545bbc6974f9f036c043ba6b8396ee86ca556c18b641198a908375dc510ff6600147304402203ab38ea5a72617a87053fae65c787488f1543da5c4307ab704a1f08f3dfc72cd022053e1ab1a0987029576656cdd591cf2291b661ca9c3cefe239f1c801565dc92e60147522102a078c9a4a92a1650e7a86e73fc1567f0c664808cb4652e307e45eb428dd454e721035f34f7c35841c2e77a3e388c36018f880dfd436e27467659b5e831be8b520d2e52ae0070a720 (11804 ms)
2019-07-25T15:02:53.979Z lightningd(6): 0279c22ed7a068d10dc1a38ae66d2d6461e269226c60258c021b1ddcdfe4b00bc4 chan #11947: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel c853101b7b61b7223e6fe196ab5819f725593f7920a76ca60bce1dd8c0af8032: sync error
2019-07-25T15:02:53.979Z lightningd(6): 	 (tx 4b9830f6e8e4dc5bcb40f80f91ad9f974a542718b9f0e4bcb543c05dbc757e89)
2019-07-25T15:02:54.138Z lightningd(6): 0279c22ed7a068d10dc1a38ae66d2d6461e269226c60258c021b1ddcdfe4b00bc4 chan #11947: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
2019-07-25T15:02:56.612Z lightningd(6): 030a7222468b178a14d46b3866a4514071650a79c950d91d0862887dcf03527f03 chan #15147: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel 40641f87ff5e078a8649174dbe649a12111f3bcd65f7c03d4f86a887981d76a2: sync error
2019-07-25T15:02:56.612Z lightningd(6): 	 (tx a3157039a7f6a0f5f426e0eb79b27e598327fdaaef3ac1cb7b4161eb4716c4a7)
2019-07-25T15:02:56.645Z lightningd(6): 030a7222468b178a14d46b3866a4514071650a79c950d91d0862887dcf03527f03 chan #15147: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
2019-07-25T15:03:03.519Z lightningd(6): 026b73b3ff65200026db80c580fa5004e79db4e13cffbcdd37cbff84bd5e16bfcd chan #7907: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel 21a4a1dca3127bc5ae0e52211a392a6ffbc478be9f949154aaeae8f89965f227: sync error
2019-07-25T15:03:03.519Z lightningd(6): 	 (tx 0452a24a2be07189fc4dbdc4aa22a61105c266d4e56cc7946b1386808cd24596)
2019-07-25T15:03:03.528Z lightningd(6): 026b73b3ff65200026db80c580fa5004e79db4e13cffbcdd37cbff84bd5e16bfcd chan #7907: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
2019-07-25T15:09:55.141Z lightningd(6): lightning_openingd-0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17858: Peer connection lost
2019-07-25T15:09:55.142Z lightningd(6): 0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17858: Owning subdaemon lightning_openingd died (62208)
2019-07-25T15:12:03.379Z lightningd(6): 02e708bcd8685044fec60699809f5e1d5523833e2c1aec3cf96dd1dedf0ba0db78 chan #17229: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-07-25T15:12:03.379Z lightningd(6): 	 (tx ffc3486aa7c568e1ee1ea22ac754dd411f823cf10966c0b3b82a15819f5fd9f5)
2019-07-25T15:12:03.382Z lightningd(6): 02e708bcd8685044fec60699809f5e1d5523833e2c1aec3cf96dd1dedf0ba0db78 chan #17229: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-07-25T15:12:03.383Z lightningd(6): 02e708bcd8685044fec60699809f5e1d5523833e2c1aec3cf96dd1dedf0ba0db78 chan #17229: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-07-25T15:12:03.392Z lightningd(6): 026b73b3ff65200026db80c580fa5004e79db4e13cffbcdd37cbff84bd5e16bfcd chan #7907: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-07-25T15:12:03.392Z lightningd(6): 	 (tx 0452a24a2be07189fc4dbdc4aa22a61105c266d4e56cc7946b1386808cd24596)
2019-07-25T15:12:03.397Z lightningd(6): 026b73b3ff65200026db80c580fa5004e79db4e13cffbcdd37cbff84bd5e16bfcd chan #7907: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-07-25T15:12:03.398Z lightningd(6): 026b73b3ff65200026db80c580fa5004e79db4e13cffbcdd37cbff84bd5e16bfcd chan #7907: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-07-25T15:12:03.407Z lightningd(6): 0279c22ed7a068d10dc1a38ae66d2d6461e269226c60258c021b1ddcdfe4b00bc4 chan #11947: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-07-25T15:12:03.407Z lightningd(6): 	 (tx 4b9830f6e8e4dc5bcb40f80f91ad9f974a542718b9f0e4bcb543c05dbc757e89)
2019-07-25T15:12:03.415Z lightningd(6): 0279c22ed7a068d10dc1a38ae66d2d6461e269226c60258c021b1ddcdfe4b00bc4 chan #11947: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-07-25T15:12:03.415Z lightningd(6): 0279c22ed7a068d10dc1a38ae66d2d6461e269226c60258c021b1ddcdfe4b00bc4 chan #11947: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-07-25T15:12:03.445Z lightningd(6): 029d3b9c46beb7303c80ab700f51652f54f472e92ffdcb6c618e8247b19827061a chan #17481: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-07-25T15:12:03.445Z lightningd(6): 	 (tx 4797a74a87f64d01b04ea82494477612ea44d947467d8d5f466f27a63e466aa8)
2019-07-25T15:12:03.461Z lightningd(6): 029d3b9c46beb7303c80ab700f51652f54f472e92ffdcb6c618e8247b19827061a chan #17481: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-07-25T15:12:03.462Z lightningd(6): 029d3b9c46beb7303c80ab700f51652f54f472e92ffdcb6c618e8247b19827061a chan #17481: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-07-25T15:12:03.487Z lightningd(6): 03a08d40fc3676e62a0bcaa27de657ce5e05b9a2491fbbf011a506a774e9ea263a chan #12073: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-07-25T15:12:03.487Z lightningd(6): 	 (tx daca27a09aa0d3b8d0256195a8ad744c9e5e1e5ddd4073b29d74040610be2a94)
2019-07-25T15:12:03.488Z lightningd(6): 03a08d40fc3676e62a0bcaa27de657ce5e05b9a2491fbbf011a506a774e9ea263a chan #12073: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-07-25T15:12:03.489Z lightningd(6): 03a08d40fc3676e62a0bcaa27de657ce5e05b9a2491fbbf011a506a774e9ea263a chan #12073: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-07-25T15:12:03.517Z lightningd(6): 02309f3556d51e9d855594762dcf68cb283fbfd7e48618b6cecb8cd90e2e34d8ad chan #2216: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-07-25T15:12:03.517Z lightningd(6): 	 (tx 3c32cd613bb2078c286c6e10860417c0c9ac796d6663763209ce3686640567a7)
2019-07-25T15:12:03.517Z lightningd(6): 02309f3556d51e9d855594762dcf68cb283fbfd7e48618b6cecb8cd90e2e34d8ad chan #2216: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-07-25T15:12:03.518Z lightningd(6): 02309f3556d51e9d855594762dcf68cb283fbfd7e48618b6cecb8cd90e2e34d8ad chan #2216: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-07-25T15:12:03.555Z lightningd(6): 0303a518845db99994783f606e6629e705cfaf072e5ce9a4d8bf9e249de4fbd019 chan #4514: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-07-25T15:12:03.555Z lightningd(6): 	 (tx 515c8b19d0291d44eb15be078e7d15380932801543668cf4799d806a8244d1af)
2019-07-25T15:12:03.556Z lightningd(6): 0303a518845db99994783f606e6629e705cfaf072e5ce9a4d8bf9e249de4fbd019 chan #4514: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-07-25T15:12:03.557Z lightningd(6): 0303a518845db99994783f606e6629e705cfaf072e5ce9a4d8bf9e249de4fbd019 chan #4514: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-07-25T15:12:03.594Z lightningd(6): 0390b5d4492dc2f5318e5233ab2cebf6d48914881a33ef6a9c6bcdbb433ad986d0 chan #4512: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-07-25T15:12:03.594Z lightningd(6): 	 (tx f2d74cfa26cb99a513ba9bebb0f3379fd9677366ea70d6c125360771e482c908)
2019-07-25T15:12:03.595Z lightningd(6): 0390b5d4492dc2f5318e5233ab2cebf6d48914881a33ef6a9c6bcdbb433ad986d0 chan #4512: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-07-25T15:12:03.596Z lightningd(6): 0390b5d4492dc2f5318e5233ab2cebf6d48914881a33ef6a9c6bcdbb433ad986d0 chan #4512: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-07-25T15:12:03.648Z lightningd(6): 030995c0c0217d763c2274aa6ed69a0bb85fa2f7d118f93631550f3b6219a577f5 chan #4513: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-07-25T15:12:03.648Z lightningd(6): 	 (tx e6ecbba710a177e40d01e136df347afe2e8da6ce44f307b1e596991ec647ed1b)
2019-07-25T15:12:03.649Z lightningd(6): 030995c0c0217d763c2274aa6ed69a0bb85fa2f7d118f93631550f3b6219a577f5 chan #4513: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-07-25T15:12:03.649Z lightningd(6): 030995c0c0217d763c2274aa6ed69a0bb85fa2f7d118f93631550f3b6219a577f5 chan #4513: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-07-25T15:12:03.695Z lightningd(6): 034ea80f8b148c750463546bd999bf7321a0e6dfc60aaf84bd0400a2e8d376c0d5 chan #7571: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-07-25T15:12:03.695Z lightningd(6): 	 (tx f601adb189d056ceb16f062f24e9cd3d4b4ee62f887f010aba945cd25e02b14d)
2019-07-25T15:12:03.696Z lightningd(6): 034ea80f8b148c750463546bd999bf7321a0e6dfc60aaf84bd0400a2e8d376c0d5 chan #7571: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-07-25T15:12:03.696Z lightningd(6): 034ea80f8b148c750463546bd999bf7321a0e6dfc60aaf84bd0400a2e8d376c0d5 chan #7571: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-07-25T15:12:03.744Z lightningd(6): 03dab87ff8635982815c4567eb58af48f9944d11c56beb12b91e1049aaea06e187 chan #7015: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-07-25T15:12:03.744Z lightningd(6): 	 (tx 3c86707e516de4badb34690f3316e4e451e9aa1e2d52afc0063adb88c9521a79)
2019-07-25T15:12:03.745Z lightningd(6): 03dab87ff8635982815c4567eb58af48f9944d11c56beb12b91e1049aaea06e187 chan #7015: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-07-25T15:12:03.752Z lightningd(6): 03dab87ff8635982815c4567eb58af48f9944d11c56beb12b91e1049aaea06e187 chan #7015: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-07-25T15:12:03.806Z lightningd(6): 03da1c27ca77872ac5b3e568af30673e599a47a5e4497f85c7b5da42048807b3ed chan #7022: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-07-25T15:12:03.806Z lightningd(6): 	 (tx 193be06dcd3a75f26545026cb13769d4bb65de0e396dd672e2cd2df413bc39b6)
2019-07-25T15:12:03.807Z lightningd(6): 03da1c27ca77872ac5b3e568af30673e599a47a5e4497f85c7b5da42048807b3ed chan #7022: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-07-25T15:12:03.807Z lightningd(6): 03da1c27ca77872ac5b3e568af30673e599a47a5e4497f85c7b5da42048807b3ed chan #7022: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-07-25T15:12:03.861Z lightningd(6): 03d37fca0656558de4fd86bbe490a38d84a46228e7ec1361801f54f9437a18d618 chan #9225: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-07-25T15:12:03.861Z lightningd(6): 	 (tx edeaec247dbba4e1399b07130349cede852c40dfecb012940b9da329dd330765)
2019-07-25T15:12:03.862Z lightningd(6): 03d37fca0656558de4fd86bbe490a38d84a46228e7ec1361801f54f9437a18d618 chan #9225: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-07-25T15:12:03.862Z lightningd(6): 03d37fca0656558de4fd86bbe490a38d84a46228e7ec1361801f54f9437a18d618 chan #9225: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-07-25T15:12:03.930Z lightningd(6): 03bc9337c7a28bb784d67742ebedd30a93bacdf7e4ca16436ef3798000242b2251 chan #9702: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-07-25T15:12:03.930Z lightningd(6): 	 (tx 6880208215eaaf23fad2bc219649d60c75c189fad9e6c6b005fb8ad3e30cdf65)
2019-07-25T15:12:03.937Z lightningd(6): 03bc9337c7a28bb784d67742ebedd30a93bacdf7e4ca16436ef3798000242b2251 chan #9702: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-07-25T15:12:03.945Z lightningd(6): 03bc9337c7a28bb784d67742ebedd30a93bacdf7e4ca16436ef3798000242b2251 chan #9702: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-07-25T15:12:04.001Z lightningd(6): 03323b53ff21a3d5d0b87c4a94e85bc65a136a93f54d255cb9069afac3b9a40169 chan #9929: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-07-25T15:12:04.001Z lightningd(6): 	 (tx 4a367dc136e476cec7401ae393bd5183f4eec1551df3ab9c172a319889ef76b4)
2019-07-25T15:12:04.002Z lightningd(6): 03323b53ff21a3d5d0b87c4a94e85bc65a136a93f54d255cb9069afac3b9a40169 chan #9929: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-07-25T15:12:04.009Z lightningd(6): 03323b53ff21a3d5d0b87c4a94e85bc65a136a93f54d255cb9069afac3b9a40169 chan #9929: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-07-25T15:12:04.105Z lightningd(6): 02c91d6aa51aa940608b497b6beebcb1aec05be3c47704b682b3889424679ca490 chan #7020: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-07-25T15:12:04.105Z lightningd(6): 	 (tx f5f8485c98d02982c6381846d22e3655e66dd5c6f671c9d1091d3bad250e05ce)
2019-07-25T15:12:04.117Z lightningd(6): 02c91d6aa51aa940608b497b6beebcb1aec05be3c47704b682b3889424679ca490 chan #7020: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-07-25T15:12:04.121Z lightningd(6): 02c91d6aa51aa940608b497b6beebcb1aec05be3c47704b682b3889424679ca490 chan #7020: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-07-25T15:12:04.182Z lightningd(6): 022c260f9ad58196af280c80a96ec9eabf6404df59ff1a7553b0f381c875a29ba0 chan #2419: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-07-25T15:12:04.182Z lightningd(6): 	 (tx 8f6f886c12520203b43279a26d7faca3b4c2d8320b6fefca8dec2f0f9526c0e7)
2019-07-25T15:12:04.205Z lightningd(6): 022c260f9ad58196af280c80a96ec9eabf6404df59ff1a7553b0f381c875a29ba0 chan #2419: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-07-25T15:12:04.205Z lightningd(6): 022c260f9ad58196af280c80a96ec9eabf6404df59ff1a7553b0f381c875a29ba0 chan #2419: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-07-25T15:12:04.286Z lightningd(6): 022ec48186f2366b429d2fca1af2f4fc33dd97a8f42c55ab4c4ab4947f8a3cf768 chan #17833: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-07-25T15:12:04.286Z lightningd(6): 	 (tx 703837e5110a282a5e4b6b8253e42d3f1f116afb9b21b6511a619b688148830d)
2019-07-25T15:12:04.287Z lightningd(6): 022ec48186f2366b429d2fca1af2f4fc33dd97a8f42c55ab4c4ab4947f8a3cf768 chan #17833: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-07-25T15:12:04.288Z lightningd(6): 022ec48186f2366b429d2fca1af2f4fc33dd97a8f42c55ab4c4ab4947f8a3cf768 chan #17833: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-07-25T15:12:04.329Z lightningd(6): 02247d9db0dfafea745ef8c9e161eb322f73ac3f8858d8730b6fd97254747ce76b chan #9681: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-07-25T15:12:04.329Z lightningd(6): 	 (tx 826052b9dc8570c5d3bbc8d20b489f5bb62075ab1a84767a3cc9a67ddf745893)
2019-07-25T15:12:04.355Z lightningd(6): 02247d9db0dfafea745ef8c9e161eb322f73ac3f8858d8730b6fd97254747ce76b chan #9681: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-07-25T15:12:04.366Z lightningd(6): 02247d9db0dfafea745ef8c9e161eb322f73ac3f8858d8730b6fd97254747ce76b chan #9681: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-07-25T15:12:04.445Z lightningd(6): 030a7222468b178a14d46b3866a4514071650a79c950d91d0862887dcf03527f03 chan #15147: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-07-25T15:12:04.445Z lightningd(6): 	 (tx a3157039a7f6a0f5f426e0eb79b27e598327fdaaef3ac1cb7b4161eb4716c4a7)
2019-07-25T15:12:04.453Z lightningd(6): 030a7222468b178a14d46b3866a4514071650a79c950d91d0862887dcf03527f03 chan #15147: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-07-25T15:12:04.454Z lightningd(6): 030a7222468b178a14d46b3866a4514071650a79c950d91d0862887dcf03527f03 chan #15147: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-07-25T15:12:04.509Z lightningd(6): 03fce165537aea120bffe8505876b44d5119354f825b3eac329b761fc5636bf334 chan #4515: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-07-25T15:12:04.509Z lightningd(6): 	 (tx e3f7c5eabb1623ab13ecf3ab1154fbdd0df98fcfd066fc2cf5cb39601c4b6c49)
2019-07-25T15:12:04.533Z lightningd(6): 03fce165537aea120bffe8505876b44d5119354f825b3eac329b761fc5636bf334 chan #4515: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-07-25T15:12:04.533Z lightningd(6): 03fce165537aea120bffe8505876b44d5119354f825b3eac329b761fc5636bf334 chan #4515: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-07-25T15:12:04.629Z lightningd(6): 024d2387409269f3b79e2708bb39b895c9f4b6a8322153af54eba487d4993bf60f chan #4517: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-07-25T15:12:04.629Z lightningd(6): 	 (tx fd39b5e41a51d8d322023f1ab1e0050e1db3f24c8aa528a1caa968d4a58d606f)
2019-07-25T15:12:04.637Z lightningd(6): 024d2387409269f3b79e2708bb39b895c9f4b6a8322153af54eba487d4993bf60f chan #4517: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-07-25T15:12:04.638Z lightningd(6): 024d2387409269f3b79e2708bb39b895c9f4b6a8322153af54eba487d4993bf60f chan #4517: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-07-25T15:12:04.725Z lightningd(6): 02725e5abcbf5550fc29e6b19706a1377f25d2b1502684f9be9965b6deac167520 chan #8461: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-07-25T15:12:04.725Z lightningd(6): 	 (tx 36a5a59342a5c4bc9684fa75447835fb84c04348869a3fc1bdec2bc1ac049dad)
2019-07-25T15:12:04.749Z lightningd(6): 02725e5abcbf5550fc29e6b19706a1377f25d2b1502684f9be9965b6deac167520 chan #8461: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-07-25T15:12:04.750Z lightningd(6): 02725e5abcbf5550fc29e6b19706a1377f25d2b1502684f9be9965b6deac167520 chan #8461: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-07-25T15:12:04.836Z lightningd(6): 02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b chan #7457: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-07-25T15:12:04.836Z lightningd(6): 	 (tx d43d014ad5754b191bf1bdaed405fa32223e6a93b0791a652da15027eddbf354)
2019-07-25T15:12:04.869Z lightningd(6): 02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b chan #7457: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-07-25T15:12:04.873Z lightningd(6): 02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b chan #7457: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-07-25T15:12:11.952Z lightningd(6): 02e708bcd8685044fec60699809f5e1d5523833e2c1aec3cf96dd1dedf0ba0db78 chan #17229: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-07-25T15:12:12.065Z lightningd(6): 026b73b3ff65200026db80c580fa5004e79db4e13cffbcdd37cbff84bd5e16bfcd chan #7907: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-07-25T15:12:12.173Z lightningd(6): 0279c22ed7a068d10dc1a38ae66d2d6461e269226c60258c021b1ddcdfe4b00bc4 chan #11947: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-07-25T15:12:12.472Z lightningd(6): 03a08d40fc3676e62a0bcaa27de657ce5e05b9a2491fbbf011a506a774e9ea263a chan #12073: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-07-25T15:12:12.952Z lightningd(6): 030995c0c0217d763c2274aa6ed69a0bb85fa2f7d118f93631550f3b6219a577f5 chan #4513: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-07-25T15:12:12.996Z lightningd(6): 029d3b9c46beb7303c80ab700f51652f54f472e92ffdcb6c618e8247b19827061a chan #17481: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-07-25T15:12:13.050Z lightningd(6): 034ea80f8b148c750463546bd999bf7321a0e6dfc60aaf84bd0400a2e8d376c0d5 chan #7571: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-07-25T15:12:13.078Z lightningd(6): 0303a518845db99994783f606e6629e705cfaf072e5ce9a4d8bf9e249de4fbd019 chan #4514: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-07-25T15:12:13.091Z lightningd(6): 02309f3556d51e9d855594762dcf68cb283fbfd7e48618b6cecb8cd90e2e34d8ad chan #2216: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-07-25T15:12:13.150Z lightningd(6): 03dab87ff8635982815c4567eb58af48f9944d11c56beb12b91e1049aaea06e187 chan #7015: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-07-25T15:12:13.205Z lightningd(6): 0390b5d4492dc2f5318e5233ab2cebf6d48914881a33ef6a9c6bcdbb433ad986d0 chan #4512: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-07-25T15:12:13.257Z lightningd(6): 03d37fca0656558de4fd86bbe490a38d84a46228e7ec1361801f54f9437a18d618 chan #9225: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-07-25T15:12:13.273Z lightningd(6): 03da1c27ca77872ac5b3e568af30673e599a47a5e4497f85c7b5da42048807b3ed chan #7022: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-07-25T15:12:13.341Z lightningd(6): 03bc9337c7a28bb784d67742ebedd30a93bacdf7e4ca16436ef3798000242b2251 chan #9702: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-07-25T15:12:13.355Z lightningd(6): 03323b53ff21a3d5d0b87c4a94e85bc65a136a93f54d255cb9069afac3b9a40169 chan #9929: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-07-25T15:12:13.449Z lightningd(6): 02c91d6aa51aa940608b497b6beebcb1aec05be3c47704b682b3889424679ca490 chan #7020: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-07-25T15:12:13.457Z lightningd(6): 022c260f9ad58196af280c80a96ec9eabf6404df59ff1a7553b0f381c875a29ba0 chan #2419: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-07-25T15:12:13.489Z lightningd(6): 022ec48186f2366b429d2fca1af2f4fc33dd97a8f42c55ab4c4ab4947f8a3cf768 chan #17833: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-07-25T15:12:13.518Z lightningd(6): 02247d9db0dfafea745ef8c9e161eb322f73ac3f8858d8730b6fd97254747ce76b chan #9681: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-07-25T15:12:13.543Z lightningd(6): 030a7222468b178a14d46b3866a4514071650a79c950d91d0862887dcf03527f03 chan #15147: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-07-25T15:12:13.574Z lightningd(6): 03fce165537aea120bffe8505876b44d5119354f825b3eac329b761fc5636bf334 chan #4515: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-07-25T15:12:13.584Z lightningd(6): 024d2387409269f3b79e2708bb39b895c9f4b6a8322153af54eba487d4993bf60f chan #4517: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-07-25T15:12:13.599Z lightningd(6): 02725e5abcbf5550fc29e6b19706a1377f25d2b1502684f9be9965b6deac167520 chan #8461: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-07-25T15:12:13.608Z lightningd(6): 02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b chan #7457: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-07-25T15:19:56.145Z lightningd(6): lightning_openingd-0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17859: Peer connection lost
2019-07-25T15:19:56.146Z lightningd(6): 0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17859: Owning subdaemon lightning_openingd died (9)
2019-07-25T15:22:54.049Z lightningd(6): lightning_openingd-0318f342854597b2a9af86ae818b794264b3c32e07626ad402a5c47b673482d105 chan #17860: Peer connection lost
2019-07-25T15:22:54.051Z lightningd(6): 0318f342854597b2a9af86ae818b794264b3c32e07626ad402a5c47b673482d105 chan #17860: Owning subdaemon lightning_openingd died (9)
2019-07-25T15:29:35.113Z lightningd(6): lightning_openingd-030a7222468b178a14d46b3866a4514071650a79c950d91d0862887dcf03527f03 chan #17856: Peer connection lost
2019-07-25T15:29:35.114Z lightningd(6): 030a7222468b178a14d46b3866a4514071650a79c950d91d0862887dcf03527f03 chan #17856: Owning subdaemon lightning_openingd died (62208)
2019-07-25T15:32:40.983Z lightningd(6): lightning_channeld-033cfb88e57bf05550bf461878e58b08a56c4f7cc972b2428e5a35a70a0d2b25c9 chan #4712: Peer connection lost
2019-07-25T15:32:40.984Z lightningd(6): 033cfb88e57bf05550bf461878e58b08a56c4f7cc972b2428e5a35a70a0d2b25c9 chan #4712: Peer transient failure in CHANNELD_SHUTTING_DOWN: lightning_channeld: Owning subdaemon lightning_channeld died (9)
2019-07-25T15:39:55.693Z lightningd(6): lightning_openingd-0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17861: Peer connection lost
2019-07-25T15:39:55.694Z lightningd(6): 0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17861: Owning subdaemon lightning_openingd died (62208)
2019-07-25T15:42:41.903Z lightningd(6): lightning_openingd-029d3b9c46beb7303c80ab700f51652f54f472e92ffdcb6c618e8247b19827061a chan #17854: Peer connection lost
2019-07-25T15:42:41.905Z lightningd(6): 029d3b9c46beb7303c80ab700f51652f54f472e92ffdcb6c618e8247b19827061a chan #17854: Owning subdaemon lightning_openingd died (62208)
2019-07-25T15:49:56.150Z lightningd(6): lightning_openingd-0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17863: Peer connection lost
2019-07-25T15:49:56.151Z lightningd(6): 0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17863: Owning subdaemon lightning_openingd died (9)
2019-07-25T15:57:31.563Z lightningd(6): lightning_openingd-0318f342854597b2a9af86ae818b794264b3c32e07626ad402a5c47b673482d105 chan #17864: Peer connection lost
2019-07-25T15:57:31.564Z lightningd(6): 0318f342854597b2a9af86ae818b794264b3c32e07626ad402a5c47b673482d105 chan #17864: Owning subdaemon lightning_openingd died (9)
2019-07-25T15:59:55.908Z lightningd(6): lightning_openingd-0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17865: Peer connection lost
2019-07-25T15:59:55.909Z lightningd(6): 0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17865: Owning subdaemon lightning_openingd died (9)
2019-07-25T16:19:03.436Z lightningd(6): lightning_openingd-02efb7d9ddb5e7c05a3b5e6fce5d919390512b4e108e9bce850b206ab39bc32268 chan #17867: Peer connection lost
2019-07-25T16:19:03.437Z lightningd(6): 02efb7d9ddb5e7c05a3b5e6fce5d919390512b4e108e9bce850b206ab39bc32268 chan #17867: Owning subdaemon lightning_openingd died (62208)
2019-07-25T16:19:56.060Z lightningd(6): lightning_openingd-0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17866: Peer connection lost
2019-07-25T16:19:56.061Z lightningd(6): 0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17866: Owning subdaemon lightning_openingd died (62208)
2019-07-25T16:35:42.775Z lightningd(6): lightning_channeld-033cfb88e57bf05550bf461878e58b08a56c4f7cc972b2428e5a35a70a0d2b25c9 chan #4712: Peer connection lost
2019-07-25T16:35:42.776Z lightningd(6): 033cfb88e57bf05550bf461878e58b08a56c4f7cc972b2428e5a35a70a0d2b25c9 chan #4712: Peer transient failure in CHANNELD_SHUTTING_DOWN: lightning_channeld: Owning subdaemon lightning_channeld died (9)
2019-07-25T16:39:55.761Z lightningd(6): lightning_openingd-0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17868: Peer connection lost
2019-07-25T16:39:55.761Z lightningd(6): 0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17868: Owning subdaemon lightning_openingd died (62208)
2019-07-25T16:58:18.006Z lightningd(6): lightning_openingd-0318f342854597b2a9af86ae818b794264b3c32e07626ad402a5c47b673482d105 chan #17870: Peer connection lost
2019-07-25T16:58:18.007Z lightningd(6): 0318f342854597b2a9af86ae818b794264b3c32e07626ad402a5c47b673482d105 chan #17870: Owning subdaemon lightning_openingd died (9)
2019-07-25T16:59:55.774Z lightningd(6): lightning_openingd-0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17869: Peer connection lost
2019-07-25T16:59:55.775Z lightningd(6): 0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17869: Owning subdaemon lightning_openingd died (62208)
2019-07-25T17:19:56.019Z lightningd(6): lightning_openingd-0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17871: Peer connection lost
2019-07-25T17:19:56.019Z lightningd(6): 0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17871: Owning subdaemon lightning_openingd died (62208)
2019-07-25T17:24:43.089Z lightningd(6): lightning_openingd-0318f342854597b2a9af86ae818b794264b3c32e07626ad402a5c47b673482d105 chan #17872: Peer connection lost
2019-07-25T17:24:43.091Z lightningd(6): 0318f342854597b2a9af86ae818b794264b3c32e07626ad402a5c47b673482d105 chan #17872: Owning subdaemon lightning_openingd died (9)
2019-07-25T17:38:44.510Z lightningd(6): lightning_channeld-033cfb88e57bf05550bf461878e58b08a56c4f7cc972b2428e5a35a70a0d2b25c9 chan #4712: Peer connection lost
2019-07-25T17:38:44.512Z lightningd(6): 033cfb88e57bf05550bf461878e58b08a56c4f7cc972b2428e5a35a70a0d2b25c9 chan #4712: Peer transient failure in CHANNELD_SHUTTING_DOWN: lightning_channeld: Owning subdaemon lightning_channeld died (9)
2019-07-25T17:39:55.757Z lightningd(6): lightning_openingd-0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17873: Peer connection lost
2019-07-25T17:39:55.758Z lightningd(6): 0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17873: Owning subdaemon lightning_openingd died (62208)
2019-07-25T17:50:36.914Z lightningd(6): lightning_openingd-02efb7d9ddb5e7c05a3b5e6fce5d919390512b4e108e9bce850b206ab39bc32268 chan #17875: Peer connection lost
2019-07-25T17:50:36.915Z lightningd(6): 02efb7d9ddb5e7c05a3b5e6fce5d919390512b4e108e9bce850b206ab39bc32268 chan #17875: Owning subdaemon lightning_openingd died (9)
2019-07-25T17:59:55.179Z lightningd(6): lightning_openingd-0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17874: Peer connection lost
2019-07-25T17:59:55.179Z lightningd(6): 0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17874: Owning subdaemon lightning_openingd died (62208)
2019-07-25T18:09:55.868Z lightningd(6): lightning_openingd-0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17876: Peer connection lost
2019-07-25T18:09:55.868Z lightningd(6): 0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17876: Owning subdaemon lightning_openingd died (62208)
2019-07-25T18:23:16.265Z lightningd(6): lightning_openingd-02efb7d9ddb5e7c05a3b5e6fce5d919390512b4e108e9bce850b206ab39bc32268 chan #17878: Peer connection lost
2019-07-25T18:23:16.266Z lightningd(6): 02efb7d9ddb5e7c05a3b5e6fce5d919390512b4e108e9bce850b206ab39bc32268 chan #17878: Owning subdaemon lightning_openingd died (9)
2019-07-25T18:29:55.754Z lightningd(6): lightning_openingd-0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17877: Peer connection lost
2019-07-25T18:29:55.755Z lightningd(6): 0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17877: Owning subdaemon lightning_openingd died (62208)
2019-07-25T18:39:55.901Z lightningd(6): lightning_openingd-0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17879: Peer connection lost
2019-07-25T18:39:55.901Z lightningd(6): 0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17879: Owning subdaemon lightning_openingd died (62208)
2019-07-25T18:41:46.194Z lightningd(6): lightning_channeld-033cfb88e57bf05550bf461878e58b08a56c4f7cc972b2428e5a35a70a0d2b25c9 chan #4712: Peer connection lost
2019-07-25T18:41:46.195Z lightningd(6): 033cfb88e57bf05550bf461878e58b08a56c4f7cc972b2428e5a35a70a0d2b25c9 chan #4712: Peer transient failure in CHANNELD_SHUTTING_DOWN: lightning_channeld: Owning subdaemon lightning_channeld died (9)
2019-07-25T18:49:55.936Z lightningd(6): lightning_openingd-0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17880: Peer connection lost
2019-07-25T18:49:55.937Z lightningd(6): 0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17880: Owning subdaemon lightning_openingd died (62208)
2019-07-25T18:55:13.453Z lightningd(6): lightning_openingd-02efb7d9ddb5e7c05a3b5e6fce5d919390512b4e108e9bce850b206ab39bc32268 chan #17881: Peer connection lost
2019-07-25T18:55:13.454Z lightningd(6): 02efb7d9ddb5e7c05a3b5e6fce5d919390512b4e108e9bce850b206ab39bc32268 chan #17881: Owning subdaemon lightning_openingd died (9)
2019-07-25T18:59:55.564Z lightningd(6): lightning_openingd-0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17882: Peer connection lost
2019-07-25T18:59:55.566Z lightningd(6): 0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17882: Owning subdaemon lightning_openingd died (9)
2019-07-25T19:09:55.622Z lightningd(6): lightning_openingd-0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17883: Peer connection lost
2019-07-25T19:09:55.623Z lightningd(6): 0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17883: Owning subdaemon lightning_openingd died (62208)
2019-07-25T19:22:09.063Z lightningd(6): lightning_openingd-02efb7d9ddb5e7c05a3b5e6fce5d919390512b4e108e9bce850b206ab39bc32268 chan #17885: Peer connection lost
2019-07-25T19:22:09.064Z lightningd(6): 02efb7d9ddb5e7c05a3b5e6fce5d919390512b4e108e9bce850b206ab39bc32268 chan #17885: Owning subdaemon lightning_openingd died (9)
2019-07-25T19:29:55.981Z lightningd(6): lightning_openingd-0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17884: Peer connection lost
2019-07-25T19:29:55.982Z lightningd(6): 0375d873f2ad784e004c9675ceeb302b6424559fc9e84d7a535a0c010ac55e3ec3 chan #17884: Owning subdaemon lightning_openingd died (62208)

getinfo output

{
   "id" : "02d35ad6428b083d63575e3b0ce33e06c507910c131970b07dc194953b43a9794c",
   "alias" : "BitcoinShirt",
   "color" : "02d35a",
   "num_peers" : 36,
   "num_pending_channels" : 0,
   "num_active_channels" : 11,
   "num_inactive_channels" : 24,
   "address" : [
      {
         "type" : "ipv4",
         "address" : "172.81.181.159",
         "port" : 9735
      },
      {
         "type" : "torv3",
         "address" : "fj2b35ny3kaxessk6su3ydbeh3ppqmxykqzbshesf43kzwgmmrjp2sid.onion",
         "port" : 9735
      }
   ],
   "binding" : [
      {
         "type" : "ipv4",
         "address" : "0.0.0.0",
         "port" : 9735
      }
   ],
   "version" : "basedon-v0.7.1",
   "blockheight" : 587008,
   "network" : "bitcoin",
   "msatoshi_fees_collected" : 51559,
   "fees_collected_msat" : "51559msat"
}
@pavlenex pavlenex changed the title Node suddenly force-closed nearly all of my channels Node suddenly force-closed 40 channels - Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel Jul 25, 2019
@ZmnSCPxj

This comment has been minimized.

Copy link
Collaborator

commented Jul 25, 2019

Will be fixed by #2842.

@rustyrussell

This comment has been minimized.

Copy link
Contributor

commented Jul 29, 2019

Reopened.

While we have created a workaround for this, I'm not convinced that LND isn't getting upset at us for some other reason. So I've funded channels with as many of these peers as I can, and am waiting to see whether they recover next time they send "sync error".

@rustyrussell rustyrussell added this to the 0.7.2 milestone Jul 29, 2019
@rustyrussell rustyrussell self-assigned this Jul 29, 2019
@cfromknecht

This comment has been minimized.

Copy link

commented Jul 30, 2019

@rustyrussell I went through our logic again to hopefully shed some light on why this might be happening. When we return "sync error", there are two major classes where either we do or do not force close. The cases where each occur:

  • force close
    • we suspect remote data loss
    • invalid commit secret
    • invalid commitment point
  • no force close
    • we suspect local data loss
    • some other unexpected error

To me it seems unlikely that all of those peers thought that they lost data at the same time, or experienced unexpected db failures at the same time.

From the logs it also appears that the CL side only closed a handful of channels, would that indicate that the remote sides closed most of these channels? If so, it seems that would mean either that there was in fact data loss (possibly a false positive if we disagree on heights) or we are disagreeing on commit secret/point.

and am waiting to see whether they recover next time they send "sync error".

AFAICT the only case in which the channel could recover is when we experience an unexpected failure. In all other cases we mark the channel borked which will prevent it from being loaded on subsequent connections.

Hope this was helpful, if there's a way for us can get logs from the remote end that would certainly help pin it down.

EDIT: can't recover from invalid commit secret because we publish unilaterally

@rustyrussell

This comment has been minimized.

Copy link
Contributor

commented Jul 31, 2019

OK, first the good news: retrying does eventually work, and the channels recover. Phew!

@cfromknecht I managed to reproduce this on my last restart: looks like I got a wave of 24 peers(!) with sync error. This is my runs-under-valgrind node, so it's amazingly slow!

Are you getting upset when we re-transmit announcement_signatures perhaps? I can look through the other peers for patterns, but it's a big log so I'd need to know what I'm looking for...

Timeline of one peer, with commentry: (peer is 02247d9db0dfafea745ef8c9e161eb322f73ac3f8858d8730b6fd97254747ce76b):

2019-07-29T06:55:09.317Z: we connect out
2019-07-29T06:55:12.146Z: we open channel, txid 2e39d320afa21d49d2f180985bdb9d2cdf95c220b8086f2c21063127a26eaed9

# Depth needed is 3, it comes in 40 minutes later:
2019-07-29T07:35:40.387Z peer_in WIRE_FUNDING_LOCKED
2019-07-29T07:36:21.019Z peer_out WIRE_FUNDING_LOCKED

# We often update the fee once we're locked in, and we do so here.
2019-07-29T07:36:22.189Z peer_out WIRE_UPDATE_FEE
2019-07-29T07:36:23.945Z peer_out WIRE_COMMITMENT_SIGNED
2019-07-29T07:36:24.023Z peer_in WIRE_REVOKE_AND_ACK
2019-07-29T07:36:26.450Z peer_in WIRE_COMMITMENT_SIGNED
2019-07-29T07:36:27.806Z peer_out WIRE_REVOKE_AND_ACK

# Peer sees block 6 before we do:
2019-07-29T08:26:01.253Z peer_in WIRE_ANNOUNCEMENT_SIGNATURES

# We update fee, because why not?
2019-07-29T08:27:03.682Z peer_out WIRE_UPDATE_FEE
2019-07-29T08:27:05.019Z peer_out WIRE_COMMITMENT_SIGNED
2019-07-29T08:27:05.102Z peer_in WIRE_REVOKE_AND_ACK

# We finally notice the new block, depth 6:
2019-07-29T08:27:07.163Z peer_out WIRE_ANNOUNCEMENT_SIGNATURES

# You finish the update fee.
2019-07-29T08:27:07.639Z peer_in WIRE_COMMITMENT_SIGNED
2019-07-29T08:27:08.204Z peer_out WIRE_REVOKE_AND_ACK

# We update fee 20 more times, all normal.

# We crash, because 2GB + 3GB swap isn't enough for valgrind
2019-07-29T16:47:44.133Z **BROKEN** lightningd(1272): bitcoin-cli exec failed: Cannot allocate memory

# Rusty pays more money to digital ocean and restarts:
2019-07-30T01:22:39.581Z we connect out
2019-07-30T01:23:28.717Z peer_out WIRE_CHANNEL_REESTABLISH
# lnd sends announcement_signatures early, we stash it.
2019-07-30T01:23:29.067Z peer_in WIRE_ANNOUNCEMENT_SIGNATURES
2019-07-30T01:23:29.524Z peer_in WIRE_CHANNEL_REESTABLISH
# peer numbers match, nothing to re-xmit. option_dataloss fields are correct.
2019-07-30T01:23:30.955Z peer_out WIRE_ANNOUNCEMENT_SIGNATURES
2019-07-30T01:23:32.533Z Peer said 'channel d9ae6ea2273106212c6f08b820c295df2c9ddb5b9880f1d2491da2af20d3392f: sync error' so we'll come back later
rustyrussell added a commit to rustyrussell/lightning that referenced this issue Jul 31, 2019
The reason lnd was sending sync error was that we were taking more than
30 seconds to send the channel_reestablish after connect.  That's
understandable on my test node under valgrind, but shouldn't happen normally.

However, it seems it has at least once,
(see ElementsProject#2847)
: space out startup so it's less likely to happen.

Suggested-by: @cfromknecht
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell added a commit to rustyrussell/lightning that referenced this issue Jul 31, 2019
The reason lnd was sending sync error was that we were taking more than
30 seconds to send the channel_reestablish after connect.  That's
understandable on my test node under valgrind, but shouldn't happen normally.

However, it seems it has at least once,
(see ElementsProject#2847)
: space out startup so it's less likely to happen.

Suggested-by: @cfromknecht
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
@rustyrussell

This comment has been minimized.

Copy link
Contributor

commented Jul 31, 2019

OK, reproduced with help from @cfromknecht . It's us taking more than 30 seconds to send the reestablish message. LND should simply hang up (since there are no "soft" errors in the protocol), but it indicates that this workaround is correct.

It's not surprising that my node (which runs under valgrind, thus is amazingly slow!) hits this limit, but it's surprising that @pavlenex hit it. I've implemented @cfromknecht suggestion that we startup slowly, see #2885

rustyrussell added a commit to rustyrussell/lightning that referenced this issue Jul 31, 2019
The reason lnd was sending sync error was that we were taking more than
30 seconds to send the channel_reestablish after connect.  That's
understandable on my test node under valgrind, but shouldn't happen normally.

However, it seems it has at least once,
(see ElementsProject#2847)
: space out startup so it's less likely to happen.

Suggested-by: @cfromknecht
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell added a commit that referenced this issue Aug 1, 2019
The reason lnd was sending sync error was that we were taking more than
30 seconds to send the channel_reestablish after connect.  That's
understandable on my test node under valgrind, but shouldn't happen normally.

However, it seems it has at least once,
(see #2847)
: space out startup so it's less likely to happen.

Suggested-by: @cfromknecht
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.