Connectd does demux part 1

[rustyrussell]

(Builds on #4972) Rebased on master

This puts connectd in the middle of every peer connection: it does the decryption/encryption and streams the gossip_store.

Changelog-None

[cdecker]

Preliminary comments I had collected before the end of year.

[rustyrussell]

OK, last three commits are new, as is 264ae92 which is a fixup.

[cdecker]

Excellent changes, just needs squashing :-)

git range-diff $(git merge-base ef04f77735a159e042e5e05cbf4f836388f02a40 origin/master)..ef04f77735a159e042e5e05cbf4f836388f02a40 $(git merge-base 502fe1064228317c87979533e77338f102217a12 origin/master)..502fe1064228317c87979533e77338f102217a12
 1:  663b3f983 =  1:  663b3f983 update mocks
 2:  cb19d2924 =  2:  cb19d2924 connectd: keep timeout timer around so we can disable it.
 3:  66bc45a68 =  3:  66bc45a68 connectd: maintain connection with peer, shuffle data.
 4:  c19d2596e =  4:  c19d2596e gossipwith: create our own internal sync_crypto functions.
 5:  90d467ab3 !  5:  44a498920 lightningd: make sure gossipd knows before we update blockheight.
    @@ gossipd/gossipd.c: static void new_blockheight(struct daemon *daemon, const u8 *
      }

      #if DEVELOPER
    -@@ gossipd/gossipd.c: static void handle_txout_reply(struct daemon *daemon, const u8 *msg)
    -   good = handle_pending_cannouncement(daemon, daemon->rstate,
    -                                       &scid, sat, outscript);
    -
    -+
    -   /* If we looking specifically for this, we no longer are. */
    -   remove_unknown_scid(daemon->seeker, &scid, good);
    -
     @@ gossipd/gossipd.c: static struct io_plan *recv_req(struct io_conn *conn,
        case WIRE_GOSSIPD_DEV_COMPACT_STORE_REPLY:
        case WIRE_GOSSIPD_GOT_ONIONMSG_TO_US:
 6:  947e054d0 =  6:  904426100 connectd: do decryption for peers.
 7:  56a53d736 =  7:  c73e2efd0 peer_io: replace crypto_sync in daemons, use normal wire messages.
 8:  8bdba3c96 =  8:  04927f198 per_peer_state: remove struct crypto_state
 9:  dedcef40e =  9:  70d52263d connectd: do dev_disconnect logic.
10:  18942d63c = 10:  c570252f6 connectd: do nagle by packet type.
 -:  --------- > 11:  264ae921f fixup! connectd: do nagle by packet type.
11:  217b63aae = 12:  479ed13a7 common: add routine for absolute timeouts (vs. relative).
12:  7ff1a3653 ! 13:  f354c9a22 connectd: serve gossip_store file for the peer.
    @@ connectd/multiplex.c: static struct io_plan *write_to_peer(struct io_conn *peer_
                                                after_final_msg);
                }

    --          /* Reset urgent flag, if set. */
    --          set_urgent_flag(peer, false);
    +-          /* Tell them to read again, */
    +-          io_wake(&peer->subd_in);
     +          /* If they want us to send gossip, do so now. */
     +          msg = maybe_from_gossip_store(NULL, peer);
     +          if (!msg) {
    -+                  /* Reset urgent flag, if set. */
    -+                  set_urgent_flag(peer, false);
    -
    --          /* Tell them to read again, */
    --          io_wake(&peer->subd_in);
     +                  /* Tell them to read again, */
     +                  io_wake(&peer->subd_in);

13:  a93ea224f = 14:  95727c0f3 subdaemons: don't stream gossip_store at all.
14:  73b996976 = 15:  644245d27 lightningd: remove per_peer_state struct.
15:  4eac6867e = 16:  6e6ae9395 connectd: remove per_peer_state in favor of keeping gossip_fd directly.
16:  e7ddcc32d = 17:  f2d66807b connectd: put more stuff into struct gossip_state.
17:  ef04f7773 = 18:  2a1317d16 connectd: get addresses from lightningd, not gossipd.
 -:  --------- > 19:  dc94ea569 various: minor cleanups from Christian's review.
 -:  --------- > 20:  a8ef64b04 connectd: drop support (unused) for @ during handshake.
 -:  --------- > 21:  502fe1064 connectd: implement @ correctly.

ACK 502fe10

[rustyrussell]

Fixup folded. And dumb non-dev inversion fixed.

Ack 4bb4ace

[cdecker]

Shepherding CI, otherwise GTG

ACK 4bb4ace

[vincenzopalazzo]

I think there is only one missing that the Github view hide, and it is at line 4056 of channeld.c, we miss an API upgrade msg_queue_new(peer, true);

We miss at list another CI run

[cdecker]

Changes look good, just a couple of flaky tests:

• test_closing_specified_destination gets stuck on l1.rpc.call('close', {'id': chan13, 'destination': addr}). close doesn't seem to want to return a result.
• test_htlc_rexmit_while_closing and test_penalty_outhtlc seem to be killing openingd instead of reporting an Unacceptable upfront_shutdown_script. The outcome is the same (no channel) but the error message was a bit more informative.

[rustyrussell]

OK, I gave in and included many of the cleanups I had hoped to defer to a followup PR. Let's see if they help or hinder the test failures.

[rustyrussell]

That... did not help :(

Since I cannot reproduce this locally, I'll be throwing debugging patches around...

[vincenzopalazzo]

ack 886bede for me because I think that the following error is unrelated. There is only some slow run here

=================================== FAILURES ===================================
_________________________ test_upgrade_statickey_fail __________________________
[gw5] linux -- Python 3.6.15 /opt/hostedtoolcache/Python/3.6.15/x64/bin/python3

node_factory = <pyln.testing.utils.NodeFactory object at 0x7f5be9cfd1d0>
executor = <concurrent.futures.thread.ThreadPoolExecutor object at 0x7f5be9cfd518>
bitcoind = <pyln.testing.utils.BitcoinD object at 0x7f5bea5e4eb8>

    @unittest.skipIf(not EXPERIMENTAL_FEATURES, "upgrade protocol not available")
    @pytest.mark.developer("dev-force-features, dev-disconnect required")
    def test_upgrade_statickey_fail(node_factory, executor, bitcoind):
        """We reconnect at all points during retransmit, and we won't upgrade."""
        l1_disconnects = ['-WIRE_COMMITMENT_SIGNED',
                          '-WIRE_REVOKE_AND_ACK']
        l2_disconnects = ['-WIRE_REVOKE_AND_ACK',
                          '-WIRE_COMMITMENT_SIGNED']
    
        l1, l2 = node_factory.line_graph(2, opts=[{'may_reconnect': True,
                                                   'dev-no-reconnect': None,
                                                   'disconnect': l1_disconnects,
                                                   'dev-force-features': ["-13", "-21"],
                                                   # Don't have feerate changes!
                                                   'feerates': (7500, 7500, 7500, 7500)},
                                                  {'may_reconnect': True,
                                                   'dev-no-reconnect': None,
                                                   'disconnect': l2_disconnects,
                                                   'dev-disable-commit-after': 1}])
    
        # This HTLC will fail
        l1.rpc.sendpay([{'msatoshi': 1000, 'id': l2.info['id'], 'delay': 5, 'channel': '1x1x1'}], '00' * 32, payment_secret='00' * 32)
    
        # Each one should cause one disconnection, no upgrade.
        for d in l1_disconnects + l2_disconnects:
            l1.daemon.wait_for_log('Peer connection lost')
            l2.daemon.wait_for_log('Peer connection lost')
            assert not l1.daemon.is_in_log('option_static_remotekey enabled')
            assert not l2.daemon.is_in_log('option_static_remotekey enabled')
            l1.rpc.connect(l2.info['id'], 'localhost', l2.port)
    
        # On the last reconnect, it retransmitted revoke_and_ack.
        l1.daemon.wait_for_log('No upgrade: we retransmitted')
        l2.daemon.wait_for_log('No upgrade: pending changes')
    
        # Now when we reconnect, despite having an HTLC, we're quiescent.
        l1.rpc.disconnect(l2.info['id'], force=True)
        l1.rpc.connect(l2.info['id'], 'localhost', l2.port)
    
>       l1.daemon.wait_for_log('option_static_remotekey enabled at 2/2')

tests/test_connection.py:3653: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
contrib/pyln-testing/pyln/testing/utils.py:310: in wait_for_log
    return self.wait_for_logs([regex], timeout)

[rustyrussell]

Ack 886bede