ONCHAIN:Owning subdaemon lightning_onchaind died

[redstorm1]

Issue and Steps to Reproduce

Channel stuck closing down, missing inputs, failed sig checks

http://paste.ubuntu.com/p/5SpqDSb5HB/

 "type": "BROKEN", 
          "time": "12.195608833", 
          "source": "lightning_onchaind-02ed613f005d8dae7e7211d628c63c963c4df35d9b56afabac39fefe9e90a89c75 chan #4:", 
          "log": "STATUS_FAIL_INTERNAL_ERROR: htlc_timeout_fee 1672 failed sigcheck  for tx 02000000017698fdb6ad5b12f964b1e721942d0f5ae8cff290da3ae9cb9c1e92ff51b9184c02000000000000000001bb06030000000000220020151afb44dd24be5f287b1399dd0cf63c62c0fde90e587647b1c07849d6dc65a80dab1300, signature 3045022100c484336113327e47ed0d4fdaba0eca49f117e98a8ae1c584929bda86a4e2eb7d0220452c93179d6c2e35234674cde56c60f3c044c88b03f5000696357826a60082cc, wscript 76a91436b3f906c15b44d86f60db0f902c4d0cdc9751c68763ac672102a7afa2906363ee562ce1b1bc73f2094a3621af05370345bb3cb73de1b717dd277c820120876475527c2103a9de3e5d78f3b253a42421c36c29ce9e8c78ffdabd061c2c8d1a5fb3e083d4a252ae67a914ab198fefb1c023a0df97a00af41a1407a9ce3a7f88ac6868"
        }, 
        {
          "type": "BROKEN", 
          "time": "12.195727460", 
          "source": "02ed613f005d8dae7e7211d628c63c963c4df35d9b56afabac39fefe9e90a89c75 chan #4:", 
          "log": "Owning subdaemon lightning_onchaind died (61952)"
        }, 

getinfo output

{
  "id": "0392419c02fbca83ae8d541cd2f48b662b2238ed8e1e92a93547c23ba32f0d64c6",
  "port": 9735,
  "address": [
    {
      "type": "ipv4",
      "address": "203.86.204.88",
      "port": 9735
    }
  ],
  "version": "v0.5.2-2016-11-21-2556-g5ff0d40",
  "blockheight": 1294677,
  "network": "testnet"
}

[52edc]

I have similar problem.

{
      "id": "02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b",
      "connected": false,
      "alias": "rompert.com🔵",
      "color": "0000ff",
      "channels": [
        {
          "state": "ONCHAIN",
          "short_channel_id": "520138:1626:1",
          "channel_id": "573861d46513d55461b0dc4696acc907b5057a606fb3589f6da18bfac55f2b12",
          "funding_txid": "132b5fc5fa8ba16d9f58b36f607a05b507c9ac9646dcb06154d51365d4613857",
          "msatoshi_to_us": 79000000,
          "msatoshi_to_us_min": 79000000,
          "msatoshi_to_us_max": 79000000,
          "msatoshi_total": 79000000,
          "dust_limit_satoshis": 546,
          "max_htlc_value_in_flight_msat": 18446744073709551615,
          "their_channel_reserve_satoshis": 0,
          "our_channel_reserve_satoshis": 790,
          "channel_reserve_satoshis": 0,
          "spendable_msatoshi": 78210000,
          "htlc_minimum_msat": 0,
          "their_to_self_delay": 144,
          "our_to_self_delay": 144,
          "to_self_delay": 144,
          "max_accepted_htlcs": 483,
          "status": [
            "CHANNELD_AWAITING_LOCKIN:Reconnected, and reestablished.",
            "CHANNELD_NORMAL:Reconnected, and reestablished.",
            "ONCHAIN:Owning subdaemon lightning_onchaind died (61952)"
          ],
          "in_payments_offered": 0,
          "in_msatoshi_offered": 0,
          "in_payments_fulfilled": 0,
          "in_msatoshi_fulfilled": 0,
          "out_payments_offered": 0,
          "out_msatoshi_offered": 0,
          "out_payments_fulfilled": 0,
          "out_msatoshi_fulfilled": 0
        }
      ]
    },

root@vmi178998:~/lightning/lightningd# ./lightningd --network=bitcoin --alias=52edcs10ma --port 9735 --ipaddr 173.212.209.141
2018-04-28T08:28:35.923Z lightningd(1019): Restarting onchaind for channel 2
2018-04-28T08:28:35.924Z lightningd(1019): 02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b chan #2: Peer permanent failure in ONCHAIN: Funding transaction spent
2018-04-28T08:28:35.924Z lightningd(1019):       (tx f89e3ef283d5f25d5a65c8fdb8db28321a2639212c751c5267dff55ae644b3ad)
2018-04-28T08:28:35.925Z lightningd(1019): 02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b chan #2: State changed from ONCHAIN to FUNDING_SPEND_SEEN
2018-04-28T08:28:35.926Z lightningd(1019): No fee estimate for Normal: basing on default fee rate
2018-04-28T08:28:35.928Z lightningd(1019): Server started with public key 03075112debc5be99ac3947ed5da179b44e7085b65d1811f573a6ed9aae46352de, alias 52edcs10ma (color #030751) and lightningd v0.5.2-2016-11-21-2606-gabf5107
2018-04-28T08:28:35.949Z lightningd(1019): 02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b chan #2: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2018-04-28T08:28:35.951Z lightningd(1019): lightning_onchaind-02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b chan #2: STATUS_FAIL_INTERNAL_ERROR: Could not find resolution for output 0
2018-04-28T08:28:35.952Z lightningd(1019): 02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b chan #2: Owning subdaemon lightning_onchaind died (61952)

Pastebin version:
https://pastebin.com/ix5UwswQ
https://pastebin.com/CG6RWeYx

Transaction has many onchain confirmations already but no funds onchain. Also, I can not use the channel funds. It stuck :(

[52edc]

After one day waiting nothing has changed :( Funds are still not available.
I tried cli/lightning-cli dev-rescan-outputs couple of times. Also restarted the whole system but nothing helps.

[yaraaer]

Any idea? Similar problem here for a week.
"ONCHAIN:Owning subdaemon lightning_onchaind died (61952)"

[WearsMyLiberty]

Same issue for a couple weeks now with a channel that we had with Bitrefill. All of the funds were on our side.

"ONCHAIN:Owning subdaemon lightning_onchaind died (61952)"

A payment was attempted and was being routed through their channel. That payment still shows as "status": "pending" today.

This is the transaction ce7c46313204c2ac43209a5fa3601d39d3f93197699a2361122612260ad25e8f that was created showing the amounts that represent the attempted payment, and the remaining balance on our side of the channel.

[yaraaer]

WearsMyLiberty: Yes, that is quite exact description of how I came to that problem. In addition, I don't keep my LN node online all the time, roughly 12 hours a day.

[WearsMyLiberty]

Through pure luck while reading over the logs I recognized this entry as a raw bitcoin transaction and was able to manually broadcast it.

`{
          "type": "DEBUG",
          "time": "35.774261948",
          "source": "lightning_onchaind-024a2e265cd66066b78a788ae615acdc84b5b0dec9efac36d7ac87513015eaf6ed chan #30:",
          "log": "Propose handling OUR_UNILATERAL/DELAYED_OUTPUT_TO_US by OUR_DELAYED_RETURN_TO_WALLET (020000000001018f5ed20a2612261261239a699731f9d3391d60a35f9a2043acc2043231467cce0000000000900000000191820300000000001600148289d9bcba260dae16c34daa34d648e96086106803473044022029bfd0e3131c0b4678e1eefd288d8f9a0bdb2e85ec12445f95a7ed63f9f04ac30220186cf268fee459ca53a7ad1a183628f4cea34f4362c3f8e7b696e8383c6c5fe201004d63210315260202a819fb9cb142ce89c0f2551734338b4766664df09f4d44a3fd4f0f6a67029000b27521024a444207fb5a8524b6bd17f1a6e2529618c87dee71e9031d7734b426db7a1c1468ac00000000) after 144 blocks"
        },`

This was successful in returning the first output of ce7c46313204c2ac43209a5fa3601d39d3f93197699a2361122612260ad25e8f back to me.

Unfortunately the other output doesn't have a log entry with a transaction in that format so I wasn't able to also broadcast it:

  {
          "type": "DEBUG",
          "time": "35.774289548",
          "source": "lightning_onchaind-024a2e265cd66066b78a788ae615acdc84b5b0dec9efac36d7ac87513015eaf6ed chan #30:",
          "log": "Tracking output 1 of ce7c46313204c2ac43209a5fa3601d39d3f93197699a2361122612260ad25e8f: OUR_UNILATERAL/OUR_HTLC"
        },
        {
          "type": "BROKEN",
          "time": "43.132920942",
          "source": "lightning_onchaind-024a2e265cd66066b78a788ae615acdc84b5b0dec9efac36d7ac87513015eaf6ed chan #30:",
          "log": "STATUS_FAIL_INTERNAL_ERROR: grind_fee failed from 1107 - 12181 for tx 02000000018f5ed20a2612261261239a699731f9d3391d60a35f9a2043acc2043231467cce01000000000000000001b8b2030000000000220020053390e7441e886abe18e419b20e853a69e8c95a640f5dabfcdfd51816ba0d6b90fb0700, inputamount 250436, signature 304402207a484eb4cd44a8c45d0412fa3514a63b40f8825043c447f13cbd25db4e89e64702201c560ed0b770dbce97499819535d2068db82dfdc2202933e10c4ecc10ec14cd7, wscript 76a9140418aebc59f31e36b363415dac0b1d8396b09c1c8763ac6721034b33fbaf63b9e69b0e796f5128760f4f6e29eafad29368981aa323b83fbcc78a7c820120876475527c2103fcdae6f2bece3985c0e65d83e1ef57380b640fb85cf1911518c692acf225528952ae67a914261978e4ce91bfa05b17703df0a10ac96559569c88ac6868, multiplier 663"
        },
        {
          "type": "DEBUG",
          "time": "43.133152742",
          "source": "lightning_onchaind-024a2e265cd66066b78a788ae615acdc84b5b0dec9efac36d7ac87513015eaf6ed chan #30:",
          "log": "Status closed, but not exited. Killing"
        },
        {
          "type": "BROKEN",
          "time": "43.138195034",
          "source": "024a2e265cd66066b78a788ae615acdc84b5b0dec9efac36d7ac87513015eaf6ed chan #30:",
          "log": "Owning subdaemon lightning_onchaind died (9)"
        },

I've been trying to figure out if there's something in any of the logs that can be put together to broadcast that second transaction but I'm honestly in over my head with that.

Also, because I was curious, a couple weeks ago I tried again and opened another channel with Bitrefill which also ended up in the same situation.

One payment was sent which was routed through Bitrefill and arrived at the destination successfully, although it still shows as "status": "pending" to me today. Another payment was sent that was not successful and also still shows as pending. Then the same thing happened as before, the channel was closed with the same errors:

{
          "type": "DEBUG",
          "time": "243.262749249",
          "source": "lightning_onchaind-024a2e265cd66066b78a788ae615acdc84b5b0dec9efac36d7ac87513015eaf6ed chan #89:",
          "log": "Tracking output 0 of 34ef130d0c3331847daedc7b2078226721ef2d50be51bf0e8ad93913059c9256: OUR_UNILATERAL/OUR_HTLC"
        },
        {
          "type": "BROKEN",
          "time": "244.510366091",
          "source": "lightning_onchaind-024a2e265cd66066b78a788ae615acdc84b5b0dec9efac36d7ac87513015eaf6ed chan #89:",
          "log": "STATUS_FAIL_INTERNAL_ERROR: grind_fee failed from 253 - 9468 for tx 020000000156929c051339d98a0ebf51be502def21672278207bdcae7d8431330c0d13ef34000000000000000000017549000000000000220020bd24f69e7e20f0b628effacb7011c69a84539cac1318c76bacf5b98269d4bac95f070800, inputamount 25082, signature 30450221008c99a4ed177133349e7656f7580482b1a2b0485d229b97ac23ed0e1ef5c771d4022058f4b45b62325da7478b6cc56f504f54a876bd30bf94d3fb15dbec9329b9e131, wscript 76a9147411cf4ba1f55a6158e071bf61d6da19770a5a2c8763ac672103c338954f720bb6caf1a5c4affdaff7e8bc4a99f00e4cbadd280b69fa27d0122f7c820120876475527c21037ee3bcd2159cdab14f57736dad33997cf89d1e50bc9685795388563d0c19876a52ae67a9146650852c81de0220f9da9c715debbb807eea468488ac6868, multiplier 663"
        },
        {
          "type": "DEBUG",
          "time": "244.510571891",
          "source": "lightning_onchaind-024a2e265cd66066b78a788ae615acdc84b5b0dec9efac36d7ac87513015eaf6ed chan #89:",
          "log": "Status closed, but not exited. Killing"
        },
        {
          "type": "BROKEN",
          "time": "244.510790991",
          "source": "024a2e265cd66066b78a788ae615acdc84b5b0dec9efac36d7ac87513015eaf6ed chan #89:",
          "log": "Owning subdaemon lightning_onchaind died (9)"
        },

The transaction it created shows the 1st output which was the payment that arrived at the final destination but still shows as pending, the 2nd output is the failed payment attempt, 3rd is our remaining channel balance, and the 4th is the balance that was on Bitrefill's side.

https://www.smartbit.com.au/tx/34ef130d0c3331847daedc7b2078226721ef2d50be51bf0e8ad93913059c9256

Full peer debug log showing both channels:

https://paste.ubuntu.com/p/FdCPWRCwFF/

[cdecker]

@rustyrussell might have some insight into why the grind_fee might have failed

[52edc]

lightning-cli getlog debug is a large file so I uploaded it here: http://patchion.co.uk/~alarmdebowa/logi2_pawel.txt

[darosior]

Hello @redstorm1, have you encountered that lately ? If not could you close the issue please :) ?