`DEVELOPER=1` assumes that the binary has been compiled with developer set to
true, which might not be the case for plugin developers. Setting this to 0 by
default has no effect in c-lightning since we always at least set it in
`config.vars` but may prevent some issues outside.

We were relying heavily on NodeFactory to do some magic before instantiating
the Node with rpc and daemon initialized, that meant that we'd have to replace
all 3 classes when customizing the node to our needs. Moving that
initialization into the node itself means that the LightningNode class now can
be swapped out and customized, without having to wire everything else through.

Quite a few of the things in the LightningNode class are tailored to their use
in the c-lightning tests, so I decided to split those customizations out into
a sub-class, and adding one more fixture that just serves the class. This
allows us to override the LightningNode implementation in our own tests, while
still having sane defaults for other users.

Needs to initialize global now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Previously, returned null if a scriptpubkey was not Segwit; now
handles encoding to Base58 for other types.

If a 'upfront_shutdown_script' was specified, show the address +
scriptpubky in `listpeers`

Changelog-added: JSON API: `listpeers` channels now include `close_to` and `close_to_addr` iff a `close_to` address was specified at channel open

--dev-force-tmp-channel-id flag takes a 64-character hex string
to use as the temporary channel id. Useful for spec tests

[ Fixed crash in non-DEVELOPER mode --RR ]
Changelog-None

Travis adds the correct PYTHONPATH, but "make check" doesn't.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

This highlights the various places we need to change.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

We add routines to decode the expected fields from both legacy and tlv
hop formats.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-changed: JSON API: `htlc_accepted` hook has `type` (currently `legacy` or `tlv`) and other fields directly inside `onion`.
Changelog-deprecated: JSON API: `htlc_accepted` hook `per_hop_v0` object deprecated, as is `short_channel_id` for the final hop.

We keep the feature bitmap on disk, so we cache this in the struct
explicitly.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

There's only one caller.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Default is legacy.  If we have future styles, new strings can be defined,
but for now it's "tlv" or "legacy".

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

For legacy, they were the same, but for TLV we care whether it's the
final hop or not.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

This means we can make sphinx_add_v0_hop static, too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

This is required for the protocol tests, which can be slow.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

This reverts commit 48728d3.

Changelog-Fixed: ElementsProject#3231 listtransactions crash

It really, really doesn't matter.  But we were dramatically reducing
our view of the network:

In my gossip_store (mainnet):
  channel_announcement: 30349
  channel_update: 55119
  node_announcment: 1783

Changelog-Fixed: No longer discard most node_announcements (fixes ElementsProject#3194)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

We now have a global constant, prefer to use it instead of having
two variables with the same utility.

==1310== Conditional jump or move depends on uninitialised value(s)
==1310==    at 0x127C7F: io_loop_with_timers (io_loop_with_timers.c:30)
==1310==    by 0x14F0E1: plugins_init (plugin.c:1019)
==1310==    by 0x12E4B1: main (lightningd.c:694)
==1310==

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Replaced by doc/lightningd-config.5.md.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

We'll use this for logging it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

A log can have a default node_id, which can be overridden on a per-entry
basis.  This changes the format of logging, so some tests need rework.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

This is ignored in subdaemons which are per-peer, but very useful for
multi-peer daemons like connectd and gossipd.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Simply better encapsulation.   We still need to expose log_entry, since the
notification hook uses it though.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

We had a separate logbook for each peer, and copy log entries above
the printable log level into the master logbook.  This didn't always
work well, since we didn't dump it on crash for example.

Keep a single global logbook instead, and remove this infrastructure.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

1. Printed form is always "[<nodeid>-]<prefix>: <string>"
2. "jcon fd %i" becomes "jsonrpc #%i".
3. "jsonrpc" log is only used once, and is removed.
4. "database" log prefix is use for db accesses.
5. "lightningd(%i)" becomes simply "lightningd" without the pid.
6. The "lightningd_" prefix is stripped from subd log prefixes, and pid removed.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-changed: Logging: formatting made uniform: [NODEID-]SUBSYSTEM: MESSAGE
Changelog-removed: `lightning_` prefixes removed from subdaemon names, including in listpeers `owner` field.

This allows finegrained logging control of particular subdaemons or
subsystems.

To do this, we defer setting the logging levels for each log object
until after early argument parsing (since e.g. "bitcoind" log object
is created early).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-changed: Options: log-level can now specify different levels for different subsystems.

This simplifies our tests, too, since we don't need a magic option to
enable io logging in subdaemons.

Note that test_bad_onion still takes too long, due to a separate minor
bug, so that's marked and left dev-only for now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Simple refcount FTW.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

1. Don't prune the last 10%.
2. Be more aggressive on pruning IO and DEBUG.
3. Account for skipped entries correctly across multiple prunes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

They added complexity, and were only used in a few places.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

The tal overhead of 5 pointers, the linked list node is 2; and we also
tal'd the string.  That's 96 bytes per entry.

Use a simple array instead, though it means more work on deletion
since each log_entry is no longer a tal object.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

We can definitely get a pong from l1 (should test be slow enough):
it's l3 we are concerned about.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Suggested-by: @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

As suggested by @cdecker, but also did a bit of minor reformatting.

I don't like the excessive indenting in our man pages though,
but that's a separate problem.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Spaces just make life a little harder for everyone.

(Plus, fix documentation: it's 'jsonrpc' not 'json' subsystem).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

I always get this wrong, then wonder why it's dying!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Particularly important when talking with modern lnd, which
will hang up on you if you don't offer feature bit 1!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

We've been sending them errors for invalid replies; instead, this works
around it.

Changelog-Added: Workaround LND's reply_channel_range issues instead of sending error.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

So far we've only handled legacy payloads, which meant we could drop the realm
byte since it was always 0x00. Once we start handling TLV payloads the first
byte, i.e., the former realm byte, is important since it gives us the length
of the payload. This is a breaking change, however I don't think there's
anyone using the `raw_payload` as of yet.

Changelog-Changed: JSON-RPC: the `raw_payload` now includes the first byte, i.e., the realm byte, of the payload as well. This allows correct decoding of a TLV payload in the plugins.

We should be using the enum here, not the length of the payload

Requesting them is done by setting BUILD=<target_arch>

Otherwise autotools (used by external dependencies like
libsodium) is not happy with setting cross-compilation
variables and may possibly lead to unexpected results.

We have a global, let's use it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Use global everywhere.  This leaks into openingd a little, too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

When we refactored bitcoin_tx to use wally_tx, this (uncompiled)
debug statement rotted. This unrots it.

We updated the protocol spec tests to verify a sig from a hash
and a private key; this updates mkcommit + mkgossip utilities
to print out the procotol compatible SIG() notation for all signatures.

--verbose will print a computed signature and more data as well.

Also adds --verbose flag to mkgossip.

Changelog-None

Fixes: ElementsProject#3269
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: gossipd crash on huge number of unknown channels.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

This leads to all sorts of problems; in particular it's incredibly
slow (days, weeks!)  if bitcoind is a long way back.  This also changes
the behaviour of a rescan argument referring to a future block: we will
also refuse to start in that case, which I think is the correct behavior.

We already ignore bitcoind if it goes backwards while we're running.

Also cover a false positive memleak.

Changelog-Fixed: If bitcoind goes backwards (e.g. reindex) refuse to start (unless forced with --rescan).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Rounds out the application of `upfront_shutdown_script`, allowing
an accepting node to specify a close_to address.

Prior to this, only the opening node could specify one.

Changelog-Added: Plugins: Allow the 'accepter' to specify an upfront_shutdown_script for a channel via a `close_to` field in the openchannel hook result

We were skipping it when running under valgrind, but not if not in developer,
which is required to have access to `dev-*` methods.

Postgresql does not guarantee insertion order when querying, so we need to
actually go and look for the correct transaction.

Signed-off-by: Christian Decker <decker.christian@gmail.com>

I missed this test breaking with `EXPERIMENTAL_FEATURES` in ElementsProject#3261

This should prevent Github from showing the diffs by default, they'd be
duplicates of the changes in the .md files anyway.

We were weaving in and out of generic code through `fromwire_tlvs` with custom
parameters defining the types in that namespace. This hard-wires the parser
with the namespace's types. Slowly trying to deprecate `fromwire_tlvs` in
favor of this typesafe variant.

Since the parser itself just parses and doesn't include validation anymore we
need to put that functionality somewhere. The validation consists of enforcing
that the types are in monotonically increasing order without duplicates and
that for the even types we know how to handle it.

We wire in the code-generated function, which removes the upfront validation
and add the validation back after the `htlc_accepted` hook returns. If a
plugin wanted to handle the onion in a special way it'll not have told us to
just continue.

We'll need to pass them around anyway, so just make them easier to access by
doing a bit more to `process_onionpacket`.

We make the fields in `htlc_accepted_payload` optional (NULL if not present in
the payload) and defer validation till after the hook call.

This now enforces all rules for validity, both for the TLV format and checking
that the required fields have been provided.

We have consolidated the two functions into a single `route_step_decode`
function, and made it static since we call it in the `process_onionpacket`
function. We remove the two exposed functions since they're no longer useful.

This function ensures we have all the infos we need to continue if the
htlc_accepted hook tells us to. It also enforces well-formedness of the TLV
payload if we have a TLV payload.

Suggested-by: List Neigut <@niftynei>
Signed-off-by: Christian Decker <@cdecker>

Travis randomly picked up an error in test_feerate_stress:
**BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-channeld-chan#1: Cannot add htlc #0 10000msat to LOCAL (version a2541b9-modded)

This is because it hit an unlikely corner case involving applying multiple HTLCs
(similar to the previous c96cee9).

In this case, the test sends a 500,000,000 "balancing" setup payment L1->L2.
It waits for L2 to get the preimage (which is the when pay() helper returns),
but crucially, it starts spamming with HTLCs before that HTLC is completely
removed.

From L2's point of view, the setup HTLC is in state RCVD_REMOVE_REVOCATION;
gone from L1's commitment tx, but still waiting for the commitment_signed
from L1 to remove it from L2's.

Note that each side keeps a local and remove view of both sides' current
balances: at this point, L2's view is REMOTE: "500,000,000 to L1, 499,900,000
to L2", LOCAL: "500,000,000 to L1, 0 to L2".

L2 sends a 10,000 msat HTLC to L1: legal, since L1 will allow it,
then the commitment_signed.  L1 sends the revoke-and-ack for this,
*then* belatedly follows with the commitment_signed which both completes the
removal of the setup HTLC and adds the new one.

But L2 processes the HTLCs in hashtable (i.e. random) order: so if it
tries to apply its own HTLC first, it freaks out because it doesn't have
funds in its local view.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Unlikely corner case is simultanous HTLCs near balance limits fixed.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Config: configuration files now support `include`.

…annel_start

With coming changes, this will segfault if we access it when param
code is trying to get usage from functions.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

lightning-cli is going to need to know what network we're on, so
it will need to parse the config files.  Move the code which does
the initial bootstrap parsing into common, as well as the config
file parsing core.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

This lets you have a default, but also a network-specific config.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-changed: Options: `config` and <network>/`config` read by default.

…twork> subdir

Changelog-changed: .lightningd plugins and files moved into <network>/ subdir
Changelog-changed: WARNING: If you don't have a config file, you now may need to specify the network to lightning-cli
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

We're going to want this for changing the default network.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Changelog-changed: Default network (new installs) is now bitcoin, not testnet.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

1. "conf" can't be specified in a configuration file.
2. "lightning-dir" can't be specified in a configuration file unless the file
   was explicitly set with --conf=.
3. "network" options can't be set in a per-network configuration file.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

They're already qualified with network name, and there's little point
moving them; it might even be dangerous if multiple are running.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Otherwise we don't print out the upgrading messages when we move things!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Don't spend too much effort on it, but this is better than running out
of memory and crashing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>