Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Never produce dust outputs #14

Closed
rustyrussell opened this issue Nov 27, 2015 · 14 comments
Closed

Never produce dust outputs #14

rustyrussell opened this issue Nov 27, 2015 · 14 comments

Comments

@rustyrussell
Copy link
Contributor

http://lists.linuxfoundation.org/pipermail/lightning-dev/2015-November/000345.html

@NicolasDorier
Copy link
Collaborator

the problem is that dust definition is not fixed it depends on mintxrelayfee which is itself floating bitcoin/bitcoin#6722 (depends on mempool state) which is why I proposed to make dust depends on fee instead of magic rule bitcoin/bitcoin#6824 the problem when I tried to implement that is that I stumbled on a circular dependency.

You may be tempted to take a fixed value to simplify the problem, but this is forgetting that the latest bump bitcoin/bitcoin#6793 was order of magnitude higher.

I fear there is no solution without doing something to decouple mintxrelayfee from dust.

@Roasbeef
Copy link

@NicolasDorier with bitcoin/bitcoin#6722, minTxRelayFee remains static. Therefore the definition of dust remains fixed (well, the default is fixed, but it's a matter of node policy).

@NicolasDorier
Copy link
Collaborator

I got confused by the title "Limit mempool by throwing away the cheapest txn and setting min relay fee to it" which I interpreted as : Set the mintxrelayfee with the cheapest tx in the mempool.

Anyway, I think it is fragile to rely on a fixed number for defining dust. I got burned by the bump to 0.00005, and I think it will happen again one day.

@rustyrussell
Copy link
Contributor Author

Nicolas Dorier notifications@github.com writes:

I got confused by the title "Limit mempool by throwing away the cheapest txn and setting min relay fee to it" which I interpreted as : Set the mintxrelayfee with the cheapest tx in the mempool.

Anyway, I think it is fragile to rely on a fixed number for defining dust. I got burned by the bump to 0.00005, and I think it will happen again one day.

Indeed. I would like a "would you accept this tx if it were signed?"
operation for bitcoin core, which we could use as a sanity check.

Meanwhile, we'll leave a fairly large margin for avoiding dust, I think.

Cheers,
Rusty.

@ABISprotocol
Copy link

NACK!

See bitcoin/bitcoin#6824 (comment)

and http://subsatoshi.org/ - originating from a discussion at ABISprotocol/ABIS#1

@rustyrussell
Copy link
Contributor Author

ABIS notifications@github.com writes:

NACK!

See bitcoin/bitcoin#6824 (comment)

This is irrelevant to the issue here. Dust outputs mean that
transactions will have difficulty traversing the bitcoin network, and
thus should not be produced.

Please also note that opening with "NACK" is uncivil and disrespectful;
which makes us less open to your ideas.

Cheers,
Rusty.

@ABISprotocol
Copy link

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hola Rusty, replying from my mail (see below) ~

Rusty Russell:

ABIS notifications@github.com writes:

NACK!

See
bitcoin/bitcoin#6824 (comment)

This is irrelevant to the issue here.

I don't agree that the above comment cited is irrelevant to the issue,
"Never produce dust outputs." As part of that comment, I noted that
@petertodd had developed a proposed system, dust-b-gone, in which
"Every dust txout is collected into a single transaction sending all
the funds to mining fees, and that transaction is signed with the
NONE|ANYONECANPAY flag." I also mentioned in my comment (via a link
to a comment in bitcoin/bitcoin issue 4079) that @gmaxwell at one time
suggested "a wallet dusting feature where you can tell the wallet to
send dust (e.g. small change and tiny txouts) to a list of user
specified donation targets with a monthly quota on how much will be
sent, and then it just automatically sends and reports on the results.
Then you get a three way benefit: the user gets a less fragmented
wallet, the network gets a less fragmented utxo set, and the recipient
of the funds gets the funds." Finally, and notably, I suggested in my
comment cited (which you claimed to be irrelevant) "that
http://abis.io should be looked at for ideas regarding how you can do
things with dust (in particular, use it as a gift rather than as a
problem)." Whether or not you agree with my perspective, it is not
irrelevant.

Dust outputs mean that transactions will have difficulty traversing
the bitcoin network, and thus should not be produced.

A simple way of assessing this would be to say that if you had no way
of properly managing dust then it would be problematic for
transactions, but if a proposal was developed (with an acceptable BIP)
to manage dust, it could be helpful. Again, dust need not be seen
simply as harmful ~ it depends on how it is managed and used.

Please also note that opening with "NACK" is uncivil and
disrespectful; which makes us less open to your ideas.

Presumably there are no speech rules for this repository. Rows of
ACKs or +1s could be seen as glaringly offensive. I suggest being
open to the form and character of how people speak or write and not
being offended by something that hits you the wrong way or that you
disagree with. Censorship (even when it doesn't exist) usually begins
with people claiming they are offended by something that they feel is
harmful.

Cheers, Rusty.

Cheerio,

Odinn

--- Reply to this email directly or view it on GitHub:
#14 (comment)
8550575


http://abis.io ~
"a protocol concept to enable decentralization
and expansion of a giving economy, and a new social good"
https://keybase.io/odinn
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJWjcTcAAoJEGxwq/inSG8CSLYH/RhNhSpw5m4MUK7qWi/hwW8N
JmDO3SrIk1jmB99RphLvzpVleuKy3dwO8Y9LQAjM6/OnpoCp16AZEtCywvWE6Q8W
+pbYbATMO7I06weCEY2sVeniaAAXQKvgLFwWCg5XBGw0p7vF+OLljZ/K+i0y4+hC
rAGbFQ2/a0Hsy8GQLbz4kwHQNqGWMKiX5Cr7Yv3uAMVbrEH0BJKWB5yupKqwJFN5
moyhS04FG0ztuOOvGrZhIXjXg/xt4JHwP4dgpA4M2qTl6U7ugss4laBtdFZxQcUZ
hc8G6OmpdUiT0B6mm33SDcqErloDRReIp/rPcs6dr+DGVf/cJliMlp70foLbMcM=
=L1at
-----END PGP SIGNATURE-----

@rustyrussell
Copy link
Contributor Author

Presumably there are no speech rules for this repository.

Yes, there are: be friendly! I have indicated that your terse "NACK" was not friendly.

Your suggestions about bitcoin are irrelevant here in the lightning repository. I will not hesitate to delete your comments from now on.

@NicolasDorier
Copy link
Collaborator

@ABISprotocol the reason why what you are linking as nothing to do with what we are talking is because you are speaking dust in the context of a wallet while we speak about dust in the context of lightning.

In the context of lightning, if one of the output happens to be dust during a pending commitment, it means that one of the two party can steal the money from the other. There is nothing to NACK, it is a problem to solve. A dangerous security issue provoked by dust.

@rustyrussell by thinking about it I think the solution is simple. If someone wants to open a channel of 10 BTC, ask him to send 10BTC + Additional BTC in the anchor transaction.

Then reject any commitment N+1 with the following condition:

Commitment N+1 have a party with a balance superior to 10 BTC AND Commitment N have a party with a balance inferior to 10 BTC.

When the channel get open, the channel opener will have 10BTC + Additional BTC.
But as soon as the threshold of Additional BTC of payment is crossed, then a "come back" will not be possible. As long as Additional BTC is significantly higher than probable dust, it should be safe.

I think this is the simplest way to deal with the problem.

@jtimon
Copy link
Contributor

jtimon commented Jan 7, 2016

Although minTxRelayFee remains static, it is configurable by users at run time and is a relay policy check, so it shouldn't be assumed that all nodes will consider the same outputs dust.
But I guess a sanity check with Bitcoin Core's default is the best you can do.

Indeed. I would like a "would you accept this tx if it were signed?" operation for bitcoin core, which we could use as a sanity check.

I've been considering a RPC call "verifyTx" only with the consensus checks. I also considered a verifyAndApproveTx RPC that would do both consensus and relay policy checks.
If there's demand for something like that I may code it for bitcoin-jt-0.12 (which will have the consensus code fully encapsulated, the relay policy code partially encapsulated and more relay/mining policy options than Bitcoin Core, among other things).

@ABISprotocol
Copy link

@NicolasDorier Hello, as I understood it, based on a brief discussion from about six months ago, subsatoshi ( see current project status at http://subsatoshi.org/ - developed by @ktorn et. al. ) was anticipated to probably use the Lightning network when payments accumulate to above-satoshi amounts, as they eventually make their way onto the actual blockchain.

@NicolasDorier I am curious on your thinking regarding dust (https://github.com/bitcoin/bitcoin/blob/v0.10.0rc3/src/primitives/transaction.h#L137), for in this issue in another repository, bitcoin/bitcoin#6824 you seemed to want to define it differently. You say here (in this repository) that there is a "dangerous security issue provoked by dust," but I suspect that the answer to this problem does not lie in never producing it... thus this discussion.. I understand what the issue is for lightning because I read the thread that @rustyrussell posted that kicked off this discussion. The payment channel doesn't close and there's an issue. So.... is there a way to calculate the amounts you'd need for the channel to successfully close before you initiate or update the commitment in lightning? That's my question.

There may of course be additional details or information that relate to this discussion in the context of subsatoshi which is why I've copied @ktorn on this.

@ktorn
Copy link
Contributor

ktorn commented Jan 15, 2016

@ABISprotocol I appreciate and sympathize with your philosophical concerns around what constitutes a dust transaction, but this is clearly not the appropriate thread for discussing those, for two reasons:

  1. it is a technical thread, not a political one;
  2. it deals with existing constraints of the underlying stack (i.e. bitcoin-core).

In short, this is not the dust you're looking for.

BTW, I like @NicolasDorier's idea and ideally long term we can have a bitcoin-core RPC call return the dust threshold at that point in time.

@ABISprotocol
Copy link

Hope this leads to a long term collaboration @NicolasDorier @ktorn

@NicolasDorier
Copy link
Collaborator

@jtimon verifyTx bitcoin/bitcoin#7552

rustyrussell added a commit to rustyrussell/lightning that referenced this issue Jan 15, 2019
Don't do this:
  (gdb) bt
  #0  0x00007f37ae667c40 in ?? () from /lib/x86_64-linux-gnu/libz.so.1
  #1  0x00007f37ae668b38 in ?? () from /lib/x86_64-linux-gnu/libz.so.1
  #2  0x00007f37ae669907 in deflate () from /lib/x86_64-linux-gnu/libz.so.1
  #3  0x00007f37ae674c65 in compress2 () from /lib/x86_64-linux-gnu/libz.so.1
  #4  0x000000000040cfe3 in zencode_scids (ctx=0xc1f118, scids=0x2599bc49 "\a\325{", len=176320) at gossipd/gossipd.c:218
  #5  0x000000000040d0b3 in encode_short_channel_ids_end (encoded=0x7fff8f98d9f0, max_bytes=65490) at gossipd/gossipd.c:236
  #6  0x000000000040dd28 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290511, number_of_blocks=8) at gossipd/gossipd.c:576
  #7  0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290511, number_of_blocks=16) at gossipd/gossipd.c:595
  #8  0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290495, number_of_blocks=32) at gossipd/gossipd.c:596
  #9  0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290495, number_of_blocks=64) at gossipd/gossipd.c:595
  #10 0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=128) at gossipd/gossipd.c:596
  #11 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=256) at gossipd/gossipd.c:595
  #12 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=512) at gossipd/gossipd.c:595
  #13 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=1024) at gossipd/gossipd.c:595
  ElementsProject#14 0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=2047) at gossipd/gossipd.c:596
  ElementsProject#15 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=4095) at gossipd/gossipd.c:595
  ElementsProject#16 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=8191) at gossipd/gossipd.c:595
  ElementsProject#17 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=16382) at gossipd/gossipd.c:595
  ElementsProject#18 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=32764) at gossipd/gossipd.c:595
  ElementsProject#19 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=65528) at gossipd/gossipd.c:595
  ElementsProject#20 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=131056) at gossipd/gossipd.c:595
  ElementsProject#21 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=262112) at gossipd/gossipd.c:595
  ElementsProject#22 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=524225) at gossipd/gossipd.c:595
  ElementsProject#23 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=1048450) at gossipd/gossipd.c:595
  ElementsProject#24 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=2096900) at gossipd/gossipd.c:595
  ElementsProject#25 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=4193801) at gossipd/gossipd.c:595
  ElementsProject#26 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=8387603) at gossipd/gossipd.c:595
  ElementsProject#27 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=16775207) at gossipd/gossipd.c:595
  ElementsProject#28 0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=33550414) at gossipd/gossipd.c:596
  ElementsProject#29 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=67100829) at gossipd/gossipd.c:595
  ElementsProject#30 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=134201659) at gossipd/gossipd.c:595
  ElementsProject#31 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=268403318) at gossipd/gossipd.c:595
  ElementsProject#32 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=536806636) at gossipd/gossipd.c:595
  ElementsProject#33 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=1073613273) at gossipd/gossipd.c:595
  ElementsProject#34 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=2147226547) at gossipd/gossipd.c:595
  ElementsProject#35 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=4294453094) at gossipd/gossipd.c:595
  ElementsProject#36 0x000000000040df26 in handle_query_channel_range (peer=0x3868fc8, msg=0x37e0678 "\001\ao\342\214\n\266\361\263r\301\246\242F\256c\367O\223\036\203e\341Z\b\234h\326\031") at gossipd/gossipd.c:625

The cause was that converting a block number to an scid truncates it
at 24 bits.  When we look through the index from (truncated number) to
(real end number) we get every channel, which is too large to encode,
so we iterate again.

This fixes both that problem, and also the issue that we'd end up
dividing into many empty sections until we get to the highest block
number.  Instead, we just tack the empty blocks on to then end of the
final query.

Reported-by: George Vaccaro
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell added a commit to rustyrussell/lightning that referenced this issue Jan 15, 2019
Don't do this:
  (gdb) bt
  #0  0x00007f37ae667c40 in ?? () from /lib/x86_64-linux-gnu/libz.so.1
  #1  0x00007f37ae668b38 in ?? () from /lib/x86_64-linux-gnu/libz.so.1
  #2  0x00007f37ae669907 in deflate () from /lib/x86_64-linux-gnu/libz.so.1
  #3  0x00007f37ae674c65 in compress2 () from /lib/x86_64-linux-gnu/libz.so.1
  #4  0x000000000040cfe3 in zencode_scids (ctx=0xc1f118, scids=0x2599bc49 "\a\325{", len=176320) at gossipd/gossipd.c:218
  #5  0x000000000040d0b3 in encode_short_channel_ids_end (encoded=0x7fff8f98d9f0, max_bytes=65490) at gossipd/gossipd.c:236
  #6  0x000000000040dd28 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290511, number_of_blocks=8) at gossipd/gossipd.c:576
  #7  0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290511, number_of_blocks=16) at gossipd/gossipd.c:595
  #8  0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290495, number_of_blocks=32) at gossipd/gossipd.c:596
  #9  0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290495, number_of_blocks=64) at gossipd/gossipd.c:595
  #10 0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=128) at gossipd/gossipd.c:596
  #11 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=256) at gossipd/gossipd.c:595
  #12 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=512) at gossipd/gossipd.c:595
  #13 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=1024) at gossipd/gossipd.c:595
  ElementsProject#14 0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=2047) at gossipd/gossipd.c:596
  ElementsProject#15 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=4095) at gossipd/gossipd.c:595
  ElementsProject#16 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=8191) at gossipd/gossipd.c:595
  ElementsProject#17 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=16382) at gossipd/gossipd.c:595
  ElementsProject#18 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=32764) at gossipd/gossipd.c:595
  ElementsProject#19 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=65528) at gossipd/gossipd.c:595
  ElementsProject#20 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=131056) at gossipd/gossipd.c:595
  ElementsProject#21 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=262112) at gossipd/gossipd.c:595
  ElementsProject#22 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=524225) at gossipd/gossipd.c:595
  ElementsProject#23 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=1048450) at gossipd/gossipd.c:595
  ElementsProject#24 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=2096900) at gossipd/gossipd.c:595
  ElementsProject#25 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=4193801) at gossipd/gossipd.c:595
  ElementsProject#26 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=8387603) at gossipd/gossipd.c:595
  ElementsProject#27 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=16775207) at gossipd/gossipd.c:595
  ElementsProject#28 0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=33550414) at gossipd/gossipd.c:596
  ElementsProject#29 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=67100829) at gossipd/gossipd.c:595
  ElementsProject#30 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=134201659) at gossipd/gossipd.c:595
  ElementsProject#31 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=268403318) at gossipd/gossipd.c:595
  ElementsProject#32 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=536806636) at gossipd/gossipd.c:595
  ElementsProject#33 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=1073613273) at gossipd/gossipd.c:595
  ElementsProject#34 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=2147226547) at gossipd/gossipd.c:595
  ElementsProject#35 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=4294453094) at gossipd/gossipd.c:595
  ElementsProject#36 0x000000000040df26 in handle_query_channel_range (peer=0x3868fc8, msg=0x37e0678 "\001\ao\342\214\n\266\361\263r\301\246\242F\256c\367O\223\036\203e\341Z\b\234h\326\031") at gossipd/gossipd.c:625

The cause was that converting a block number to an scid truncates it
at 24 bits.  When we look through the index from (truncated number) to
(real end number) we get every channel, which is too large to encode,
so we iterate again.

This fixes both that problem, and also the issue that we'd end up
dividing into many empty sections until we get to the highest block
number.  Instead, we just tack the empty blocks on to then end of the
final query.

(My initial version requested 0xFFFFFFFE blocks, but the dev code
which records what blocks were returned can't make a bitmap that big
on 32 bit).

Reported-by: George Vaccaro
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
cdecker pushed a commit to rustyrussell/lightning that referenced this issue Jan 15, 2019
Don't do this:
  (gdb) bt
  #0  0x00007f37ae667c40 in ?? () from /lib/x86_64-linux-gnu/libz.so.1
  #1  0x00007f37ae668b38 in ?? () from /lib/x86_64-linux-gnu/libz.so.1
  #2  0x00007f37ae669907 in deflate () from /lib/x86_64-linux-gnu/libz.so.1
  #3  0x00007f37ae674c65 in compress2 () from /lib/x86_64-linux-gnu/libz.so.1
  #4  0x000000000040cfe3 in zencode_scids (ctx=0xc1f118, scids=0x2599bc49 "\a\325{", len=176320) at gossipd/gossipd.c:218
  #5  0x000000000040d0b3 in encode_short_channel_ids_end (encoded=0x7fff8f98d9f0, max_bytes=65490) at gossipd/gossipd.c:236
  #6  0x000000000040dd28 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290511, number_of_blocks=8) at gossipd/gossipd.c:576
  #7  0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290511, number_of_blocks=16) at gossipd/gossipd.c:595
  #8  0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290495, number_of_blocks=32) at gossipd/gossipd.c:596
  #9  0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290495, number_of_blocks=64) at gossipd/gossipd.c:595
  #10 0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=128) at gossipd/gossipd.c:596
  #11 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=256) at gossipd/gossipd.c:595
  #12 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=512) at gossipd/gossipd.c:595
  #13 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=1024) at gossipd/gossipd.c:595
  ElementsProject#14 0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=2047) at gossipd/gossipd.c:596
  ElementsProject#15 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=4095) at gossipd/gossipd.c:595
  ElementsProject#16 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=8191) at gossipd/gossipd.c:595
  ElementsProject#17 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=16382) at gossipd/gossipd.c:595
  ElementsProject#18 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=32764) at gossipd/gossipd.c:595
  ElementsProject#19 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=65528) at gossipd/gossipd.c:595
  ElementsProject#20 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=131056) at gossipd/gossipd.c:595
  ElementsProject#21 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=262112) at gossipd/gossipd.c:595
  ElementsProject#22 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=524225) at gossipd/gossipd.c:595
  ElementsProject#23 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=1048450) at gossipd/gossipd.c:595
  ElementsProject#24 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=2096900) at gossipd/gossipd.c:595
  ElementsProject#25 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=4193801) at gossipd/gossipd.c:595
  ElementsProject#26 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=8387603) at gossipd/gossipd.c:595
  ElementsProject#27 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=16775207) at gossipd/gossipd.c:595
  ElementsProject#28 0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=33550414) at gossipd/gossipd.c:596
  ElementsProject#29 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=67100829) at gossipd/gossipd.c:595
  ElementsProject#30 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=134201659) at gossipd/gossipd.c:595
  ElementsProject#31 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=268403318) at gossipd/gossipd.c:595
  ElementsProject#32 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=536806636) at gossipd/gossipd.c:595
  ElementsProject#33 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=1073613273) at gossipd/gossipd.c:595
  ElementsProject#34 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=2147226547) at gossipd/gossipd.c:595
  ElementsProject#35 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=4294453094) at gossipd/gossipd.c:595
  ElementsProject#36 0x000000000040df26 in handle_query_channel_range (peer=0x3868fc8, msg=0x37e0678 "\001\ao\342\214\n\266\361\263r\301\246\242F\256c\367O\223\036\203e\341Z\b\234h\326\031") at gossipd/gossipd.c:625

The cause was that converting a block number to an scid truncates it
at 24 bits.  When we look through the index from (truncated number) to
(real end number) we get every channel, which is too large to encode,
so we iterate again.

This fixes both that problem, and also the issue that we'd end up
dividing into many empty sections until we get to the highest block
number.  Instead, we just tack the empty blocks on to then end of the
final query.

(My initial version requested 0xFFFFFFFE blocks, but the dev code
which records what blocks were returned can't make a bitmap that big
on 32 bit).

Reported-by: George Vaccaro
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
cdecker pushed a commit to rustyrussell/lightning that referenced this issue Jan 15, 2019
Don't do this:
  (gdb) bt
  #0  0x00007f37ae667c40 in ?? () from /lib/x86_64-linux-gnu/libz.so.1
  #1  0x00007f37ae668b38 in ?? () from /lib/x86_64-linux-gnu/libz.so.1
  #2  0x00007f37ae669907 in deflate () from /lib/x86_64-linux-gnu/libz.so.1
  #3  0x00007f37ae674c65 in compress2 () from /lib/x86_64-linux-gnu/libz.so.1
  #4  0x000000000040cfe3 in zencode_scids (ctx=0xc1f118, scids=0x2599bc49 "\a\325{", len=176320) at gossipd/gossipd.c:218
  #5  0x000000000040d0b3 in encode_short_channel_ids_end (encoded=0x7fff8f98d9f0, max_bytes=65490) at gossipd/gossipd.c:236
  #6  0x000000000040dd28 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290511, number_of_blocks=8) at gossipd/gossipd.c:576
  #7  0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290511, number_of_blocks=16) at gossipd/gossipd.c:595
  #8  0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290495, number_of_blocks=32) at gossipd/gossipd.c:596
  #9  0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290495, number_of_blocks=64) at gossipd/gossipd.c:595
  #10 0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=128) at gossipd/gossipd.c:596
  #11 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=256) at gossipd/gossipd.c:595
  #12 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=512) at gossipd/gossipd.c:595
  #13 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=1024) at gossipd/gossipd.c:595
  ElementsProject#14 0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=2047) at gossipd/gossipd.c:596
  ElementsProject#15 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=4095) at gossipd/gossipd.c:595
  ElementsProject#16 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=8191) at gossipd/gossipd.c:595
  ElementsProject#17 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=16382) at gossipd/gossipd.c:595
  ElementsProject#18 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=32764) at gossipd/gossipd.c:595
  ElementsProject#19 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=65528) at gossipd/gossipd.c:595
  ElementsProject#20 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=131056) at gossipd/gossipd.c:595
  ElementsProject#21 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=262112) at gossipd/gossipd.c:595
  ElementsProject#22 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=524225) at gossipd/gossipd.c:595
  ElementsProject#23 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=1048450) at gossipd/gossipd.c:595
  ElementsProject#24 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=2096900) at gossipd/gossipd.c:595
  ElementsProject#25 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=4193801) at gossipd/gossipd.c:595
  ElementsProject#26 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=8387603) at gossipd/gossipd.c:595
  ElementsProject#27 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=16775207) at gossipd/gossipd.c:595
  ElementsProject#28 0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=33550414) at gossipd/gossipd.c:596
  ElementsProject#29 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=67100829) at gossipd/gossipd.c:595
  ElementsProject#30 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=134201659) at gossipd/gossipd.c:595
  ElementsProject#31 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=268403318) at gossipd/gossipd.c:595
  ElementsProject#32 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=536806636) at gossipd/gossipd.c:595
  ElementsProject#33 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=1073613273) at gossipd/gossipd.c:595
  ElementsProject#34 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=2147226547) at gossipd/gossipd.c:595
  ElementsProject#35 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=4294453094) at gossipd/gossipd.c:595
  ElementsProject#36 0x000000000040df26 in handle_query_channel_range (peer=0x3868fc8, msg=0x37e0678 "\001\ao\342\214\n\266\361\263r\301\246\242F\256c\367O\223\036\203e\341Z\b\234h\326\031") at gossipd/gossipd.c:625

The cause was that converting a block number to an scid truncates it
at 24 bits.  When we look through the index from (truncated number) to
(real end number) we get every channel, which is too large to encode,
so we iterate again.

This fixes both that problem, and also the issue that we'd end up
dividing into many empty sections until we get to the highest block
number.  Instead, we just tack the empty blocks on to then end of the
final query.

(My initial version requested 0xFFFFFFFE blocks, but the dev code
which records what blocks were returned can't make a bitmap that big
on 32 bit).

Reported-by: George Vaccaro
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
niftynei pushed a commit that referenced this issue Jan 15, 2019
Don't do this:
  (gdb) bt
  #0  0x00007f37ae667c40 in ?? () from /lib/x86_64-linux-gnu/libz.so.1
  #1  0x00007f37ae668b38 in ?? () from /lib/x86_64-linux-gnu/libz.so.1
  #2  0x00007f37ae669907 in deflate () from /lib/x86_64-linux-gnu/libz.so.1
  #3  0x00007f37ae674c65 in compress2 () from /lib/x86_64-linux-gnu/libz.so.1
  #4  0x000000000040cfe3 in zencode_scids (ctx=0xc1f118, scids=0x2599bc49 "\a\325{", len=176320) at gossipd/gossipd.c:218
  #5  0x000000000040d0b3 in encode_short_channel_ids_end (encoded=0x7fff8f98d9f0, max_bytes=65490) at gossipd/gossipd.c:236
  #6  0x000000000040dd28 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290511, number_of_blocks=8) at gossipd/gossipd.c:576
  #7  0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290511, number_of_blocks=16) at gossipd/gossipd.c:595
  #8  0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290495, number_of_blocks=32) at gossipd/gossipd.c:596
  #9  0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290495, number_of_blocks=64) at gossipd/gossipd.c:595
  #10 0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=128) at gossipd/gossipd.c:596
  #11 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=256) at gossipd/gossipd.c:595
  #12 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=512) at gossipd/gossipd.c:595
  #13 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=1024) at gossipd/gossipd.c:595
  #14 0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=2047) at gossipd/gossipd.c:596
  #15 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=4095) at gossipd/gossipd.c:595
  #16 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=8191) at gossipd/gossipd.c:595
  #17 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=16382) at gossipd/gossipd.c:595
  #18 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=32764) at gossipd/gossipd.c:595
  #19 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=65528) at gossipd/gossipd.c:595
  #20 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=131056) at gossipd/gossipd.c:595
  #21 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=262112) at gossipd/gossipd.c:595
  #22 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=524225) at gossipd/gossipd.c:595
  #23 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=1048450) at gossipd/gossipd.c:595
  #24 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=2096900) at gossipd/gossipd.c:595
  #25 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=4193801) at gossipd/gossipd.c:595
  #26 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=8387603) at gossipd/gossipd.c:595
  #27 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=16775207) at gossipd/gossipd.c:595
  #28 0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=33550414) at gossipd/gossipd.c:596
  #29 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=67100829) at gossipd/gossipd.c:595
  #30 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=134201659) at gossipd/gossipd.c:595
  #31 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=268403318) at gossipd/gossipd.c:595
  #32 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=536806636) at gossipd/gossipd.c:595
  #33 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=1073613273) at gossipd/gossipd.c:595
  #34 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=2147226547) at gossipd/gossipd.c:595
  #35 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=4294453094) at gossipd/gossipd.c:595
  #36 0x000000000040df26 in handle_query_channel_range (peer=0x3868fc8, msg=0x37e0678 "\001\ao\342\214\n\266\361\263r\301\246\242F\256c\367O\223\036\203e\341Z\b\234h\326\031") at gossipd/gossipd.c:625

The cause was that converting a block number to an scid truncates it
at 24 bits.  When we look through the index from (truncated number) to
(real end number) we get every channel, which is too large to encode,
so we iterate again.

This fixes both that problem, and also the issue that we'd end up
dividing into many empty sections until we get to the highest block
number.  Instead, we just tack the empty blocks on to then end of the
final query.

(My initial version requested 0xFFFFFFFE blocks, but the dev code
which records what blocks were returned can't make a bitmap that big
on 32 bit).

Reported-by: George Vaccaro
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
NicolasDorier pushed a commit to btcpayserver/lightning that referenced this issue Jan 17, 2019
Don't do this:
  (gdb) bt
  #0  0x00007f37ae667c40 in ?? () from /lib/x86_64-linux-gnu/libz.so.1
  #1  0x00007f37ae668b38 in ?? () from /lib/x86_64-linux-gnu/libz.so.1
  #2  0x00007f37ae669907 in deflate () from /lib/x86_64-linux-gnu/libz.so.1
  #3  0x00007f37ae674c65 in compress2 () from /lib/x86_64-linux-gnu/libz.so.1
  #4  0x000000000040cfe3 in zencode_scids (ctx=0xc1f118, scids=0x2599bc49 "\a\325{", len=176320) at gossipd/gossipd.c:218
  ElementsProject#5  0x000000000040d0b3 in encode_short_channel_ids_end (encoded=0x7fff8f98d9f0, max_bytes=65490) at gossipd/gossipd.c:236
  ElementsProject#6  0x000000000040dd28 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290511, number_of_blocks=8) at gossipd/gossipd.c:576
  ElementsProject#7  0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290511, number_of_blocks=16) at gossipd/gossipd.c:595
  ElementsProject#8  0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290495, number_of_blocks=32) at gossipd/gossipd.c:596
  ElementsProject#9  0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290495, number_of_blocks=64) at gossipd/gossipd.c:595
  ElementsProject#10 0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=128) at gossipd/gossipd.c:596
  ElementsProject#11 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=256) at gossipd/gossipd.c:595
  ElementsProject#12 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=512) at gossipd/gossipd.c:595
  ElementsProject#13 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=1024) at gossipd/gossipd.c:595
  ElementsProject#14 0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=2047) at gossipd/gossipd.c:596
  ElementsProject#15 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=4095) at gossipd/gossipd.c:595
  ElementsProject#16 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=8191) at gossipd/gossipd.c:595
  ElementsProject#17 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=16382) at gossipd/gossipd.c:595
  ElementsProject#18 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=32764) at gossipd/gossipd.c:595
  ElementsProject#19 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=65528) at gossipd/gossipd.c:595
  ElementsProject#20 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=131056) at gossipd/gossipd.c:595
  ElementsProject#21 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=262112) at gossipd/gossipd.c:595
  ElementsProject#22 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=524225) at gossipd/gossipd.c:595
  ElementsProject#23 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=1048450) at gossipd/gossipd.c:595
  ElementsProject#24 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=2096900) at gossipd/gossipd.c:595
  ElementsProject#25 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=4193801) at gossipd/gossipd.c:595
  ElementsProject#26 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=8387603) at gossipd/gossipd.c:595
  ElementsProject#27 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=16775207) at gossipd/gossipd.c:595
  ElementsProject#28 0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=33550414) at gossipd/gossipd.c:596
  ElementsProject#29 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=67100829) at gossipd/gossipd.c:595
  ElementsProject#30 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=134201659) at gossipd/gossipd.c:595
  ElementsProject#31 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=268403318) at gossipd/gossipd.c:595
  ElementsProject#32 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=536806636) at gossipd/gossipd.c:595
  ElementsProject#33 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=1073613273) at gossipd/gossipd.c:595
  ElementsProject#34 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=2147226547) at gossipd/gossipd.c:595
  ElementsProject#35 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=4294453094) at gossipd/gossipd.c:595
  ElementsProject#36 0x000000000040df26 in handle_query_channel_range (peer=0x3868fc8, msg=0x37e0678 "\001\ao\342\214\n\266\361\263r\301\246\242F\256c\367O\223\036\203e\341Z\b\234h\326\031") at gossipd/gossipd.c:625

The cause was that converting a block number to an scid truncates it
at 24 bits.  When we look through the index from (truncated number) to
(real end number) we get every channel, which is too large to encode,
so we iterate again.

This fixes both that problem, and also the issue that we'd end up
dividing into many empty sections until we get to the highest block
number.  Instead, we just tack the empty blocks on to then end of the
final query.

(My initial version requested 0xFFFFFFFE blocks, but the dev code
which records what blocks were returned can't make a bitmap that big
on 32 bit).

Reported-by: George Vaccaro
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
danielsocials pushed a commit to lightningcn/lightning that referenced this issue Feb 15, 2019
Don't do this:
  (gdb) bt
  #0  0x00007f37ae667c40 in ?? () from /lib/x86_64-linux-gnu/libz.so.1
  ElementsProject#1  0x00007f37ae668b38 in ?? () from /lib/x86_64-linux-gnu/libz.so.1
  ElementsProject#2  0x00007f37ae669907 in deflate () from /lib/x86_64-linux-gnu/libz.so.1
  ElementsProject#3  0x00007f37ae674c65 in compress2 () from /lib/x86_64-linux-gnu/libz.so.1
  ElementsProject#4  0x000000000040cfe3 in zencode_scids (ctx=0xc1f118, scids=0x2599bc49 "\a\325{", len=176320) at gossipd/gossipd.c:218
  ElementsProject#5  0x000000000040d0b3 in encode_short_channel_ids_end (encoded=0x7fff8f98d9f0, max_bytes=65490) at gossipd/gossipd.c:236
  ElementsProject#6  0x000000000040dd28 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290511, number_of_blocks=8) at gossipd/gossipd.c:576
  ElementsProject#7  0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290511, number_of_blocks=16) at gossipd/gossipd.c:595
  ElementsProject#8  0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290495, number_of_blocks=32) at gossipd/gossipd.c:596
  ElementsProject#9  0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290495, number_of_blocks=64) at gossipd/gossipd.c:595
  ElementsProject#10 0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=128) at gossipd/gossipd.c:596
  ElementsProject#11 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=256) at gossipd/gossipd.c:595
  ElementsProject#12 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=512) at gossipd/gossipd.c:595
  ElementsProject#13 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17290431, number_of_blocks=1024) at gossipd/gossipd.c:595
  ElementsProject#14 0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=2047) at gossipd/gossipd.c:596
  ElementsProject#15 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=4095) at gossipd/gossipd.c:595
  ElementsProject#16 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=8191) at gossipd/gossipd.c:595
  ElementsProject#17 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=16382) at gossipd/gossipd.c:595
  ElementsProject#18 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=32764) at gossipd/gossipd.c:595
  ElementsProject#19 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=65528) at gossipd/gossipd.c:595
  ElementsProject#20 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=131056) at gossipd/gossipd.c:595
  ElementsProject#21 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=262112) at gossipd/gossipd.c:595
  ElementsProject#22 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=524225) at gossipd/gossipd.c:595
  ElementsProject#23 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=1048450) at gossipd/gossipd.c:595
  ElementsProject#24 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=2096900) at gossipd/gossipd.c:595
  ElementsProject#25 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=4193801) at gossipd/gossipd.c:595
  ElementsProject#26 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=8387603) at gossipd/gossipd.c:595
  ElementsProject#27 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=17289408, number_of_blocks=16775207) at gossipd/gossipd.c:595
  ElementsProject#28 0x000000000040ddee in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=33550414) at gossipd/gossipd.c:596
  ElementsProject#29 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=67100829) at gossipd/gossipd.c:595
  ElementsProject#30 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=134201659) at gossipd/gossipd.c:595
  ElementsProject#31 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=268403318) at gossipd/gossipd.c:595
  ElementsProject#32 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=536806636) at gossipd/gossipd.c:595
  ElementsProject#33 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=1073613273) at gossipd/gossipd.c:595
  ElementsProject#34 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=2147226547) at gossipd/gossipd.c:595
  ElementsProject#35 0x000000000040ddc6 in queue_channel_ranges (peer=0x3868fc8, first_blocknum=514201, number_of_blocks=4294453094) at gossipd/gossipd.c:595
  ElementsProject#36 0x000000000040df26 in handle_query_channel_range (peer=0x3868fc8, msg=0x37e0678 "\001\ao\342\214\n\266\361\263r\301\246\242F\256c\367O\223\036\203e\341Z\b\234h\326\031") at gossipd/gossipd.c:625

The cause was that converting a block number to an scid truncates it
at 24 bits.  When we look through the index from (truncated number) to
(real end number) we get every channel, which is too large to encode,
so we iterate again.

This fixes both that problem, and also the issue that we'd end up
dividing into many empty sections until we get to the highest block
number.  Instead, we just tack the empty blocks on to then end of the
final query.

(My initial version requested 0xFFFFFFFE blocks, but the dev code
which records what blocks were returned can't make a bitmap that big
on 32 bit).

Reported-by: George Vaccaro
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell added a commit to rustyrussell/lightning that referenced this issue Jun 12, 2019
Direct leak of 64 byte(s) in 1 object(s) allocated from:
    #0 0x7f4dc279163e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x564ee8a24bb1 in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x564ee8a2551b in double_table ccan/ccan/htable/htable.c:226
    #3 0x564ee8a259e5 in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x564ee89a5300 in block_map_add lightningd/chaintopology.h:83
    #5 0x564ee89a6ece in add_tip lightningd/chaintopology.c:626
    #6 0x564ee89a72c3 in have_new_block lightningd/chaintopology.c:694
    #7 0x564ee89a3ab0 in process_rawblock lightningd/bitcoind.c:466
    #8 0x564ee89a2fb4 in bcli_finished lightningd/bitcoind.c:214
    #9 0x564ee8a284d6 in destroy_conn ccan/ccan/io/poll.c:244
    #10 0x564ee8a284f6 in destroy_conn_close_fd ccan/ccan/io/poll.c:250
    #11 0x564ee8a34a0d in notify ccan/ccan/tal/tal.c:235
    #12 0x564ee8a34efc in del_tree ccan/ccan/tal/tal.c:397
    #13 0x564ee8a35288 in tal_free ccan/ccan/tal/tal.c:481
    ElementsProject#14 0x564ee8a26cf5 in io_close ccan/ccan/io/io.c:450
    ElementsProject#15 0x564ee8a28c11 in io_loop ccan/ccan/io/poll.c:449
    ElementsProject#16 0x564ee89b3c3b in io_loop_with_timers lightningd/io_loop_with_timers.c:24
    ElementsProject#17 0x564ee89ba540 in main lightningd/lightningd.c:822
    ElementsProject#18 0x7f4dc2143b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell added a commit to rustyrussell/lightning that referenced this issue Jun 12, 2019
Direct leak of 32 byte(s) in 1 object(s) allocated from:
    #0 0x7f7678ee863e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x55f8c7b0fce5 in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x55f8c7b1064f in double_table ccan/ccan/htable/htable.c:226
    #3 0x55f8c7b10b19 in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x55f8c7afac63 in scriptpubkeyset_add wallet/txfilter.c:30
    #5 0x55f8c7afafce in txfilter_add_scriptpubkey wallet/txfilter.c:77
    #6 0x55f8c7afb05f in txfilter_add_derkey wallet/txfilter.c:91
    #7 0x55f8c7aa4d67 in init_txfilter lightningd/lightningd.c:482
    #8 0x55f8c7aa52d8 in main lightningd/lightningd.c:721
    #9 0x7f767889ab6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7f05f389563e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x55cac1e6bc99 in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x55cac1e6c603 in double_table ccan/ccan/htable/htable.c:226
    #3 0x55cac1e6cacd in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x55cac1e56e48 in outpointset_add wallet/txfilter.c:61
    #5 0x55cac1e57162 in outpointfilter_add wallet/txfilter.c:116
    #6 0x55cac1e5ea3a in wallet_utxoset_add wallet/wallet.c:2365
    #7 0x55cac1deddc2 in topo_add_utxos lightningd/chaintopology.c:603
    #8 0x55cac1dedeac in add_tip lightningd/chaintopology.c:620
    #9 0x55cac1dee2de in have_new_block lightningd/chaintopology.c:694
    #10 0x55cac1deaab0 in process_rawblock lightningd/bitcoind.c:466
    #11 0x55cac1de9fb4 in bcli_finished lightningd/bitcoind.c:214
    #12 0x55cac1e6f5be in destroy_conn ccan/ccan/io/poll.c:244
    #13 0x55cac1e6f5de in destroy_conn_close_fd ccan/ccan/io/poll.c:250
    ElementsProject#14 0x55cac1e7baf5 in notify ccan/ccan/tal/tal.c:235
    ElementsProject#15 0x55cac1e7bfe4 in del_tree ccan/ccan/tal/tal.c:397
    ElementsProject#16 0x55cac1e7c370 in tal_free ccan/ccan/tal/tal.c:481
    ElementsProject#17 0x55cac1e6dddd in io_close ccan/ccan/io/io.c:450
    ElementsProject#18 0x55cac1e6fcf9 in io_loop ccan/ccan/io/poll.c:449
    ElementsProject#19 0x55cac1dfac66 in io_loop_with_timers lightningd/io_loop_with_timers.c:24
    ElementsProject#20 0x55cac1e0156b in main lightningd/lightningd.c:822
    ElementsProject#21 0x7f05f3247b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell added a commit to rustyrussell/lightning that referenced this issue Jun 12, 2019
Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7ff02889063e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x555ce2ad8d2e in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x555ce2ad9698 in double_table ccan/ccan/htable/htable.c:226
    #3 0x555ce2ad9b62 in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x555ce2a638e4 in htlc_in_map_add lightningd/htlc_end.h:113
    #5 0x555ce2a63beb in connect_htlc_in lightningd/htlc_end.c:39
    #6 0x555ce2a85cbc in channel_added_their_htlc lightningd/peer_htlcs.c:1382
    #7 0x555ce2a860e1 in peer_got_commitsig lightningd/peer_htlcs.c:1466
    #8 0x555ce2a5db04 in channel_msg lightningd/channel_control.c:228
    #9 0x555ce2a8d393 in sd_msg_read lightningd/subd.c:474
    #10 0x555ce2ada157 in next_plan ccan/ccan/io/io.c:59
    #11 0x555ce2adacd4 in do_plan ccan/ccan/io/io.c:407
    #12 0x555ce2adad12 in io_ready ccan/ccan/io/io.c:417
    #13 0x555ce2adcd67 in io_loop ccan/ccan/io/poll.c:445
    ElementsProject#14 0x555ce2a67c66 in io_loop_with_timers lightningd/io_loop_with_timers.c:24
    ElementsProject#15 0x555ce2a6e56b in main lightningd/lightningd.c:822
    ElementsProject#16 0x7ff028242b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell added a commit to rustyrussell/lightning that referenced this issue Jun 12, 2019
Direct leak of 64 byte(s) in 1 object(s) allocated from:
    #0 0x7f4dc279163e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x564ee8a24bb1 in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x564ee8a2551b in double_table ccan/ccan/htable/htable.c:226
    #3 0x564ee8a259e5 in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x564ee89a5300 in block_map_add lightningd/chaintopology.h:83
    #5 0x564ee89a6ece in add_tip lightningd/chaintopology.c:626
    #6 0x564ee89a72c3 in have_new_block lightningd/chaintopology.c:694
    #7 0x564ee89a3ab0 in process_rawblock lightningd/bitcoind.c:466
    #8 0x564ee89a2fb4 in bcli_finished lightningd/bitcoind.c:214
    #9 0x564ee8a284d6 in destroy_conn ccan/ccan/io/poll.c:244
    #10 0x564ee8a284f6 in destroy_conn_close_fd ccan/ccan/io/poll.c:250
    #11 0x564ee8a34a0d in notify ccan/ccan/tal/tal.c:235
    #12 0x564ee8a34efc in del_tree ccan/ccan/tal/tal.c:397
    #13 0x564ee8a35288 in tal_free ccan/ccan/tal/tal.c:481
    ElementsProject#14 0x564ee8a26cf5 in io_close ccan/ccan/io/io.c:450
    ElementsProject#15 0x564ee8a28c11 in io_loop ccan/ccan/io/poll.c:449
    ElementsProject#16 0x564ee89b3c3b in io_loop_with_timers lightningd/io_loop_with_timers.c:24
    ElementsProject#17 0x564ee89ba540 in main lightningd/lightningd.c:822
    ElementsProject#18 0x7f4dc2143b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell added a commit to rustyrussell/lightning that referenced this issue Jun 12, 2019
Direct leak of 32 byte(s) in 1 object(s) allocated from:
    #0 0x7f7678ee863e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x55f8c7b0fce5 in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x55f8c7b1064f in double_table ccan/ccan/htable/htable.c:226
    #3 0x55f8c7b10b19 in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x55f8c7afac63 in scriptpubkeyset_add wallet/txfilter.c:30
    #5 0x55f8c7afafce in txfilter_add_scriptpubkey wallet/txfilter.c:77
    #6 0x55f8c7afb05f in txfilter_add_derkey wallet/txfilter.c:91
    #7 0x55f8c7aa4d67 in init_txfilter lightningd/lightningd.c:482
    #8 0x55f8c7aa52d8 in main lightningd/lightningd.c:721
    #9 0x7f767889ab6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7f05f389563e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x55cac1e6bc99 in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x55cac1e6c603 in double_table ccan/ccan/htable/htable.c:226
    #3 0x55cac1e6cacd in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x55cac1e56e48 in outpointset_add wallet/txfilter.c:61
    #5 0x55cac1e57162 in outpointfilter_add wallet/txfilter.c:116
    #6 0x55cac1e5ea3a in wallet_utxoset_add wallet/wallet.c:2365
    #7 0x55cac1deddc2 in topo_add_utxos lightningd/chaintopology.c:603
    #8 0x55cac1dedeac in add_tip lightningd/chaintopology.c:620
    #9 0x55cac1dee2de in have_new_block lightningd/chaintopology.c:694
    #10 0x55cac1deaab0 in process_rawblock lightningd/bitcoind.c:466
    #11 0x55cac1de9fb4 in bcli_finished lightningd/bitcoind.c:214
    #12 0x55cac1e6f5be in destroy_conn ccan/ccan/io/poll.c:244
    #13 0x55cac1e6f5de in destroy_conn_close_fd ccan/ccan/io/poll.c:250
    ElementsProject#14 0x55cac1e7baf5 in notify ccan/ccan/tal/tal.c:235
    ElementsProject#15 0x55cac1e7bfe4 in del_tree ccan/ccan/tal/tal.c:397
    ElementsProject#16 0x55cac1e7c370 in tal_free ccan/ccan/tal/tal.c:481
    ElementsProject#17 0x55cac1e6dddd in io_close ccan/ccan/io/io.c:450
    ElementsProject#18 0x55cac1e6fcf9 in io_loop ccan/ccan/io/poll.c:449
    ElementsProject#19 0x55cac1dfac66 in io_loop_with_timers lightningd/io_loop_with_timers.c:24
    ElementsProject#20 0x55cac1e0156b in main lightningd/lightningd.c:822
    ElementsProject#21 0x7f05f3247b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell added a commit to rustyrussell/lightning that referenced this issue Jun 12, 2019
Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7ff02889063e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x555ce2ad8d2e in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x555ce2ad9698 in double_table ccan/ccan/htable/htable.c:226
    #3 0x555ce2ad9b62 in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x555ce2a638e4 in htlc_in_map_add lightningd/htlc_end.h:113
    #5 0x555ce2a63beb in connect_htlc_in lightningd/htlc_end.c:39
    #6 0x555ce2a85cbc in channel_added_their_htlc lightningd/peer_htlcs.c:1382
    #7 0x555ce2a860e1 in peer_got_commitsig lightningd/peer_htlcs.c:1466
    #8 0x555ce2a5db04 in channel_msg lightningd/channel_control.c:228
    #9 0x555ce2a8d393 in sd_msg_read lightningd/subd.c:474
    #10 0x555ce2ada157 in next_plan ccan/ccan/io/io.c:59
    #11 0x555ce2adacd4 in do_plan ccan/ccan/io/io.c:407
    #12 0x555ce2adad12 in io_ready ccan/ccan/io/io.c:417
    #13 0x555ce2adcd67 in io_loop ccan/ccan/io/poll.c:445
    ElementsProject#14 0x555ce2a67c66 in io_loop_with_timers lightningd/io_loop_with_timers.c:24
    ElementsProject#15 0x555ce2a6e56b in main lightningd/lightningd.c:822
    ElementsProject#16 0x7ff028242b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell added a commit to rustyrussell/lightning that referenced this issue Jun 12, 2019
Direct leak of 64 byte(s) in 1 object(s) allocated from:
    #0 0x7f4dc279163e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x564ee8a24bb1 in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x564ee8a2551b in double_table ccan/ccan/htable/htable.c:226
    #3 0x564ee8a259e5 in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x564ee89a5300 in block_map_add lightningd/chaintopology.h:83
    #5 0x564ee89a6ece in add_tip lightningd/chaintopology.c:626
    #6 0x564ee89a72c3 in have_new_block lightningd/chaintopology.c:694
    #7 0x564ee89a3ab0 in process_rawblock lightningd/bitcoind.c:466
    #8 0x564ee89a2fb4 in bcli_finished lightningd/bitcoind.c:214
    #9 0x564ee8a284d6 in destroy_conn ccan/ccan/io/poll.c:244
    #10 0x564ee8a284f6 in destroy_conn_close_fd ccan/ccan/io/poll.c:250
    #11 0x564ee8a34a0d in notify ccan/ccan/tal/tal.c:235
    #12 0x564ee8a34efc in del_tree ccan/ccan/tal/tal.c:397
    #13 0x564ee8a35288 in tal_free ccan/ccan/tal/tal.c:481
    ElementsProject#14 0x564ee8a26cf5 in io_close ccan/ccan/io/io.c:450
    ElementsProject#15 0x564ee8a28c11 in io_loop ccan/ccan/io/poll.c:449
    ElementsProject#16 0x564ee89b3c3b in io_loop_with_timers lightningd/io_loop_with_timers.c:24
    ElementsProject#17 0x564ee89ba540 in main lightningd/lightningd.c:822
    ElementsProject#18 0x7f4dc2143b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell added a commit to rustyrussell/lightning that referenced this issue Jun 12, 2019
Direct leak of 32 byte(s) in 1 object(s) allocated from:
    #0 0x7f7678ee863e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x55f8c7b0fce5 in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x55f8c7b1064f in double_table ccan/ccan/htable/htable.c:226
    #3 0x55f8c7b10b19 in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x55f8c7afac63 in scriptpubkeyset_add wallet/txfilter.c:30
    #5 0x55f8c7afafce in txfilter_add_scriptpubkey wallet/txfilter.c:77
    #6 0x55f8c7afb05f in txfilter_add_derkey wallet/txfilter.c:91
    #7 0x55f8c7aa4d67 in init_txfilter lightningd/lightningd.c:482
    #8 0x55f8c7aa52d8 in main lightningd/lightningd.c:721
    #9 0x7f767889ab6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7f05f389563e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x55cac1e6bc99 in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x55cac1e6c603 in double_table ccan/ccan/htable/htable.c:226
    #3 0x55cac1e6cacd in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x55cac1e56e48 in outpointset_add wallet/txfilter.c:61
    #5 0x55cac1e57162 in outpointfilter_add wallet/txfilter.c:116
    #6 0x55cac1e5ea3a in wallet_utxoset_add wallet/wallet.c:2365
    #7 0x55cac1deddc2 in topo_add_utxos lightningd/chaintopology.c:603
    #8 0x55cac1dedeac in add_tip lightningd/chaintopology.c:620
    #9 0x55cac1dee2de in have_new_block lightningd/chaintopology.c:694
    #10 0x55cac1deaab0 in process_rawblock lightningd/bitcoind.c:466
    #11 0x55cac1de9fb4 in bcli_finished lightningd/bitcoind.c:214
    #12 0x55cac1e6f5be in destroy_conn ccan/ccan/io/poll.c:244
    #13 0x55cac1e6f5de in destroy_conn_close_fd ccan/ccan/io/poll.c:250
    ElementsProject#14 0x55cac1e7baf5 in notify ccan/ccan/tal/tal.c:235
    ElementsProject#15 0x55cac1e7bfe4 in del_tree ccan/ccan/tal/tal.c:397
    ElementsProject#16 0x55cac1e7c370 in tal_free ccan/ccan/tal/tal.c:481
    ElementsProject#17 0x55cac1e6dddd in io_close ccan/ccan/io/io.c:450
    ElementsProject#18 0x55cac1e6fcf9 in io_loop ccan/ccan/io/poll.c:449
    ElementsProject#19 0x55cac1dfac66 in io_loop_with_timers lightningd/io_loop_with_timers.c:24
    ElementsProject#20 0x55cac1e0156b in main lightningd/lightningd.c:822
    ElementsProject#21 0x7f05f3247b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell added a commit to rustyrussell/lightning that referenced this issue Jun 12, 2019
Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7ff02889063e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x555ce2ad8d2e in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x555ce2ad9698 in double_table ccan/ccan/htable/htable.c:226
    #3 0x555ce2ad9b62 in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x555ce2a638e4 in htlc_in_map_add lightningd/htlc_end.h:113
    #5 0x555ce2a63beb in connect_htlc_in lightningd/htlc_end.c:39
    #6 0x555ce2a85cbc in channel_added_their_htlc lightningd/peer_htlcs.c:1382
    #7 0x555ce2a860e1 in peer_got_commitsig lightningd/peer_htlcs.c:1466
    #8 0x555ce2a5db04 in channel_msg lightningd/channel_control.c:228
    #9 0x555ce2a8d393 in sd_msg_read lightningd/subd.c:474
    #10 0x555ce2ada157 in next_plan ccan/ccan/io/io.c:59
    #11 0x555ce2adacd4 in do_plan ccan/ccan/io/io.c:407
    #12 0x555ce2adad12 in io_ready ccan/ccan/io/io.c:417
    #13 0x555ce2adcd67 in io_loop ccan/ccan/io/poll.c:445
    ElementsProject#14 0x555ce2a67c66 in io_loop_with_timers lightningd/io_loop_with_timers.c:24
    ElementsProject#15 0x555ce2a6e56b in main lightningd/lightningd.c:822
    ElementsProject#16 0x7ff028242b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell added a commit to rustyrussell/lightning that referenced this issue Jun 14, 2019
Direct leak of 64 byte(s) in 1 object(s) allocated from:
    #0 0x7f4dc279163e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x564ee8a24bb1 in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x564ee8a2551b in double_table ccan/ccan/htable/htable.c:226
    #3 0x564ee8a259e5 in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x564ee89a5300 in block_map_add lightningd/chaintopology.h:83
    #5 0x564ee89a6ece in add_tip lightningd/chaintopology.c:626
    #6 0x564ee89a72c3 in have_new_block lightningd/chaintopology.c:694
    #7 0x564ee89a3ab0 in process_rawblock lightningd/bitcoind.c:466
    #8 0x564ee89a2fb4 in bcli_finished lightningd/bitcoind.c:214
    #9 0x564ee8a284d6 in destroy_conn ccan/ccan/io/poll.c:244
    #10 0x564ee8a284f6 in destroy_conn_close_fd ccan/ccan/io/poll.c:250
    #11 0x564ee8a34a0d in notify ccan/ccan/tal/tal.c:235
    #12 0x564ee8a34efc in del_tree ccan/ccan/tal/tal.c:397
    #13 0x564ee8a35288 in tal_free ccan/ccan/tal/tal.c:481
    ElementsProject#14 0x564ee8a26cf5 in io_close ccan/ccan/io/io.c:450
    ElementsProject#15 0x564ee8a28c11 in io_loop ccan/ccan/io/poll.c:449
    ElementsProject#16 0x564ee89b3c3b in io_loop_with_timers lightningd/io_loop_with_timers.c:24
    ElementsProject#17 0x564ee89ba540 in main lightningd/lightningd.c:822
    ElementsProject#18 0x7f4dc2143b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell added a commit to rustyrussell/lightning that referenced this issue Jun 14, 2019
Direct leak of 32 byte(s) in 1 object(s) allocated from:
    #0 0x7f7678ee863e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x55f8c7b0fce5 in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x55f8c7b1064f in double_table ccan/ccan/htable/htable.c:226
    #3 0x55f8c7b10b19 in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x55f8c7afac63 in scriptpubkeyset_add wallet/txfilter.c:30
    #5 0x55f8c7afafce in txfilter_add_scriptpubkey wallet/txfilter.c:77
    #6 0x55f8c7afb05f in txfilter_add_derkey wallet/txfilter.c:91
    #7 0x55f8c7aa4d67 in init_txfilter lightningd/lightningd.c:482
    #8 0x55f8c7aa52d8 in main lightningd/lightningd.c:721
    #9 0x7f767889ab6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7f05f389563e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x55cac1e6bc99 in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x55cac1e6c603 in double_table ccan/ccan/htable/htable.c:226
    #3 0x55cac1e6cacd in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x55cac1e56e48 in outpointset_add wallet/txfilter.c:61
    #5 0x55cac1e57162 in outpointfilter_add wallet/txfilter.c:116
    #6 0x55cac1e5ea3a in wallet_utxoset_add wallet/wallet.c:2365
    #7 0x55cac1deddc2 in topo_add_utxos lightningd/chaintopology.c:603
    #8 0x55cac1dedeac in add_tip lightningd/chaintopology.c:620
    #9 0x55cac1dee2de in have_new_block lightningd/chaintopology.c:694
    #10 0x55cac1deaab0 in process_rawblock lightningd/bitcoind.c:466
    #11 0x55cac1de9fb4 in bcli_finished lightningd/bitcoind.c:214
    #12 0x55cac1e6f5be in destroy_conn ccan/ccan/io/poll.c:244
    #13 0x55cac1e6f5de in destroy_conn_close_fd ccan/ccan/io/poll.c:250
    ElementsProject#14 0x55cac1e7baf5 in notify ccan/ccan/tal/tal.c:235
    ElementsProject#15 0x55cac1e7bfe4 in del_tree ccan/ccan/tal/tal.c:397
    ElementsProject#16 0x55cac1e7c370 in tal_free ccan/ccan/tal/tal.c:481
    ElementsProject#17 0x55cac1e6dddd in io_close ccan/ccan/io/io.c:450
    ElementsProject#18 0x55cac1e6fcf9 in io_loop ccan/ccan/io/poll.c:449
    ElementsProject#19 0x55cac1dfac66 in io_loop_with_timers lightningd/io_loop_with_timers.c:24
    ElementsProject#20 0x55cac1e0156b in main lightningd/lightningd.c:822
    ElementsProject#21 0x7f05f3247b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell added a commit to rustyrussell/lightning that referenced this issue Jun 14, 2019
Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7ff02889063e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x555ce2ad8d2e in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x555ce2ad9698 in double_table ccan/ccan/htable/htable.c:226
    #3 0x555ce2ad9b62 in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x555ce2a638e4 in htlc_in_map_add lightningd/htlc_end.h:113
    #5 0x555ce2a63beb in connect_htlc_in lightningd/htlc_end.c:39
    #6 0x555ce2a85cbc in channel_added_their_htlc lightningd/peer_htlcs.c:1382
    #7 0x555ce2a860e1 in peer_got_commitsig lightningd/peer_htlcs.c:1466
    #8 0x555ce2a5db04 in channel_msg lightningd/channel_control.c:228
    #9 0x555ce2a8d393 in sd_msg_read lightningd/subd.c:474
    #10 0x555ce2ada157 in next_plan ccan/ccan/io/io.c:59
    #11 0x555ce2adacd4 in do_plan ccan/ccan/io/io.c:407
    #12 0x555ce2adad12 in io_ready ccan/ccan/io/io.c:417
    #13 0x555ce2adcd67 in io_loop ccan/ccan/io/poll.c:445
    ElementsProject#14 0x555ce2a67c66 in io_loop_with_timers lightningd/io_loop_with_timers.c:24
    ElementsProject#15 0x555ce2a6e56b in main lightningd/lightningd.c:822
    ElementsProject#16 0x7ff028242b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell added a commit to rustyrussell/lightning that referenced this issue Jun 14, 2019
Direct leak of 64 byte(s) in 1 object(s) allocated from:
    #0 0x7f4dc279163e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x564ee8a24bb1 in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x564ee8a2551b in double_table ccan/ccan/htable/htable.c:226
    #3 0x564ee8a259e5 in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x564ee89a5300 in block_map_add lightningd/chaintopology.h:83
    #5 0x564ee89a6ece in add_tip lightningd/chaintopology.c:626
    #6 0x564ee89a72c3 in have_new_block lightningd/chaintopology.c:694
    #7 0x564ee89a3ab0 in process_rawblock lightningd/bitcoind.c:466
    #8 0x564ee89a2fb4 in bcli_finished lightningd/bitcoind.c:214
    #9 0x564ee8a284d6 in destroy_conn ccan/ccan/io/poll.c:244
    #10 0x564ee8a284f6 in destroy_conn_close_fd ccan/ccan/io/poll.c:250
    #11 0x564ee8a34a0d in notify ccan/ccan/tal/tal.c:235
    #12 0x564ee8a34efc in del_tree ccan/ccan/tal/tal.c:397
    #13 0x564ee8a35288 in tal_free ccan/ccan/tal/tal.c:481
    ElementsProject#14 0x564ee8a26cf5 in io_close ccan/ccan/io/io.c:450
    ElementsProject#15 0x564ee8a28c11 in io_loop ccan/ccan/io/poll.c:449
    ElementsProject#16 0x564ee89b3c3b in io_loop_with_timers lightningd/io_loop_with_timers.c:24
    ElementsProject#17 0x564ee89ba540 in main lightningd/lightningd.c:822
    ElementsProject#18 0x7f4dc2143b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell added a commit to rustyrussell/lightning that referenced this issue Jun 14, 2019
Direct leak of 32 byte(s) in 1 object(s) allocated from:
    #0 0x7f7678ee863e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x55f8c7b0fce5 in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x55f8c7b1064f in double_table ccan/ccan/htable/htable.c:226
    #3 0x55f8c7b10b19 in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x55f8c7afac63 in scriptpubkeyset_add wallet/txfilter.c:30
    #5 0x55f8c7afafce in txfilter_add_scriptpubkey wallet/txfilter.c:77
    #6 0x55f8c7afb05f in txfilter_add_derkey wallet/txfilter.c:91
    #7 0x55f8c7aa4d67 in init_txfilter lightningd/lightningd.c:482
    #8 0x55f8c7aa52d8 in main lightningd/lightningd.c:721
    #9 0x7f767889ab6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7f05f389563e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x55cac1e6bc99 in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x55cac1e6c603 in double_table ccan/ccan/htable/htable.c:226
    #3 0x55cac1e6cacd in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x55cac1e56e48 in outpointset_add wallet/txfilter.c:61
    #5 0x55cac1e57162 in outpointfilter_add wallet/txfilter.c:116
    #6 0x55cac1e5ea3a in wallet_utxoset_add wallet/wallet.c:2365
    #7 0x55cac1deddc2 in topo_add_utxos lightningd/chaintopology.c:603
    #8 0x55cac1dedeac in add_tip lightningd/chaintopology.c:620
    #9 0x55cac1dee2de in have_new_block lightningd/chaintopology.c:694
    #10 0x55cac1deaab0 in process_rawblock lightningd/bitcoind.c:466
    #11 0x55cac1de9fb4 in bcli_finished lightningd/bitcoind.c:214
    #12 0x55cac1e6f5be in destroy_conn ccan/ccan/io/poll.c:244
    #13 0x55cac1e6f5de in destroy_conn_close_fd ccan/ccan/io/poll.c:250
    ElementsProject#14 0x55cac1e7baf5 in notify ccan/ccan/tal/tal.c:235
    ElementsProject#15 0x55cac1e7bfe4 in del_tree ccan/ccan/tal/tal.c:397
    ElementsProject#16 0x55cac1e7c370 in tal_free ccan/ccan/tal/tal.c:481
    ElementsProject#17 0x55cac1e6dddd in io_close ccan/ccan/io/io.c:450
    ElementsProject#18 0x55cac1e6fcf9 in io_loop ccan/ccan/io/poll.c:449
    ElementsProject#19 0x55cac1dfac66 in io_loop_with_timers lightningd/io_loop_with_timers.c:24
    ElementsProject#20 0x55cac1e0156b in main lightningd/lightningd.c:822
    ElementsProject#21 0x7f05f3247b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell added a commit to rustyrussell/lightning that referenced this issue Jun 14, 2019
Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7ff02889063e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x555ce2ad8d2e in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x555ce2ad9698 in double_table ccan/ccan/htable/htable.c:226
    #3 0x555ce2ad9b62 in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x555ce2a638e4 in htlc_in_map_add lightningd/htlc_end.h:113
    #5 0x555ce2a63beb in connect_htlc_in lightningd/htlc_end.c:39
    #6 0x555ce2a85cbc in channel_added_their_htlc lightningd/peer_htlcs.c:1382
    #7 0x555ce2a860e1 in peer_got_commitsig lightningd/peer_htlcs.c:1466
    #8 0x555ce2a5db04 in channel_msg lightningd/channel_control.c:228
    #9 0x555ce2a8d393 in sd_msg_read lightningd/subd.c:474
    #10 0x555ce2ada157 in next_plan ccan/ccan/io/io.c:59
    #11 0x555ce2adacd4 in do_plan ccan/ccan/io/io.c:407
    #12 0x555ce2adad12 in io_ready ccan/ccan/io/io.c:417
    #13 0x555ce2adcd67 in io_loop ccan/ccan/io/poll.c:445
    ElementsProject#14 0x555ce2a67c66 in io_loop_with_timers lightningd/io_loop_with_timers.c:24
    ElementsProject#15 0x555ce2a6e56b in main lightningd/lightningd.c:822
    ElementsProject#16 0x7ff028242b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell added a commit that referenced this issue Jun 30, 2019
Direct leak of 64 byte(s) in 1 object(s) allocated from:
    #0 0x7f4dc279163e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x564ee8a24bb1 in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x564ee8a2551b in double_table ccan/ccan/htable/htable.c:226
    #3 0x564ee8a259e5 in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x564ee89a5300 in block_map_add lightningd/chaintopology.h:83
    #5 0x564ee89a6ece in add_tip lightningd/chaintopology.c:626
    #6 0x564ee89a72c3 in have_new_block lightningd/chaintopology.c:694
    #7 0x564ee89a3ab0 in process_rawblock lightningd/bitcoind.c:466
    #8 0x564ee89a2fb4 in bcli_finished lightningd/bitcoind.c:214
    #9 0x564ee8a284d6 in destroy_conn ccan/ccan/io/poll.c:244
    #10 0x564ee8a284f6 in destroy_conn_close_fd ccan/ccan/io/poll.c:250
    #11 0x564ee8a34a0d in notify ccan/ccan/tal/tal.c:235
    #12 0x564ee8a34efc in del_tree ccan/ccan/tal/tal.c:397
    #13 0x564ee8a35288 in tal_free ccan/ccan/tal/tal.c:481
    #14 0x564ee8a26cf5 in io_close ccan/ccan/io/io.c:450
    #15 0x564ee8a28c11 in io_loop ccan/ccan/io/poll.c:449
    #16 0x564ee89b3c3b in io_loop_with_timers lightningd/io_loop_with_timers.c:24
    #17 0x564ee89ba540 in main lightningd/lightningd.c:822
    #18 0x7f4dc2143b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell added a commit that referenced this issue Jun 30, 2019
Direct leak of 32 byte(s) in 1 object(s) allocated from:
    #0 0x7f7678ee863e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x55f8c7b0fce5 in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x55f8c7b1064f in double_table ccan/ccan/htable/htable.c:226
    #3 0x55f8c7b10b19 in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x55f8c7afac63 in scriptpubkeyset_add wallet/txfilter.c:30
    #5 0x55f8c7afafce in txfilter_add_scriptpubkey wallet/txfilter.c:77
    #6 0x55f8c7afb05f in txfilter_add_derkey wallet/txfilter.c:91
    #7 0x55f8c7aa4d67 in init_txfilter lightningd/lightningd.c:482
    #8 0x55f8c7aa52d8 in main lightningd/lightningd.c:721
    #9 0x7f767889ab6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7f05f389563e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x55cac1e6bc99 in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x55cac1e6c603 in double_table ccan/ccan/htable/htable.c:226
    #3 0x55cac1e6cacd in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x55cac1e56e48 in outpointset_add wallet/txfilter.c:61
    #5 0x55cac1e57162 in outpointfilter_add wallet/txfilter.c:116
    #6 0x55cac1e5ea3a in wallet_utxoset_add wallet/wallet.c:2365
    #7 0x55cac1deddc2 in topo_add_utxos lightningd/chaintopology.c:603
    #8 0x55cac1dedeac in add_tip lightningd/chaintopology.c:620
    #9 0x55cac1dee2de in have_new_block lightningd/chaintopology.c:694
    #10 0x55cac1deaab0 in process_rawblock lightningd/bitcoind.c:466
    #11 0x55cac1de9fb4 in bcli_finished lightningd/bitcoind.c:214
    #12 0x55cac1e6f5be in destroy_conn ccan/ccan/io/poll.c:244
    #13 0x55cac1e6f5de in destroy_conn_close_fd ccan/ccan/io/poll.c:250
    #14 0x55cac1e7baf5 in notify ccan/ccan/tal/tal.c:235
    #15 0x55cac1e7bfe4 in del_tree ccan/ccan/tal/tal.c:397
    #16 0x55cac1e7c370 in tal_free ccan/ccan/tal/tal.c:481
    #17 0x55cac1e6dddd in io_close ccan/ccan/io/io.c:450
    #18 0x55cac1e6fcf9 in io_loop ccan/ccan/io/poll.c:449
    #19 0x55cac1dfac66 in io_loop_with_timers lightningd/io_loop_with_timers.c:24
    #20 0x55cac1e0156b in main lightningd/lightningd.c:822
    #21 0x7f05f3247b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell added a commit that referenced this issue Jun 30, 2019
Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7ff02889063e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x555ce2ad8d2e in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x555ce2ad9698 in double_table ccan/ccan/htable/htable.c:226
    #3 0x555ce2ad9b62 in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x555ce2a638e4 in htlc_in_map_add lightningd/htlc_end.h:113
    #5 0x555ce2a63beb in connect_htlc_in lightningd/htlc_end.c:39
    #6 0x555ce2a85cbc in channel_added_their_htlc lightningd/peer_htlcs.c:1382
    #7 0x555ce2a860e1 in peer_got_commitsig lightningd/peer_htlcs.c:1466
    #8 0x555ce2a5db04 in channel_msg lightningd/channel_control.c:228
    #9 0x555ce2a8d393 in sd_msg_read lightningd/subd.c:474
    #10 0x555ce2ada157 in next_plan ccan/ccan/io/io.c:59
    #11 0x555ce2adacd4 in do_plan ccan/ccan/io/io.c:407
    #12 0x555ce2adad12 in io_ready ccan/ccan/io/io.c:417
    #13 0x555ce2adcd67 in io_loop ccan/ccan/io/poll.c:445
    #14 0x555ce2a67c66 in io_loop_with_timers lightningd/io_loop_with_timers.c:24
    #15 0x555ce2a6e56b in main lightningd/lightningd.c:822
    #16 0x7ff028242b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell added a commit that referenced this issue Jun 30, 2019
Direct leak of 64 byte(s) in 1 object(s) allocated from:
    #0 0x7f4dc279163e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x564ee8a24bb1 in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x564ee8a2551b in double_table ccan/ccan/htable/htable.c:226
    #3 0x564ee8a259e5 in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x564ee89a5300 in block_map_add lightningd/chaintopology.h:83
    #5 0x564ee89a6ece in add_tip lightningd/chaintopology.c:626
    #6 0x564ee89a72c3 in have_new_block lightningd/chaintopology.c:694
    #7 0x564ee89a3ab0 in process_rawblock lightningd/bitcoind.c:466
    #8 0x564ee89a2fb4 in bcli_finished lightningd/bitcoind.c:214
    #9 0x564ee8a284d6 in destroy_conn ccan/ccan/io/poll.c:244
    #10 0x564ee8a284f6 in destroy_conn_close_fd ccan/ccan/io/poll.c:250
    #11 0x564ee8a34a0d in notify ccan/ccan/tal/tal.c:235
    #12 0x564ee8a34efc in del_tree ccan/ccan/tal/tal.c:397
    #13 0x564ee8a35288 in tal_free ccan/ccan/tal/tal.c:481
    #14 0x564ee8a26cf5 in io_close ccan/ccan/io/io.c:450
    #15 0x564ee8a28c11 in io_loop ccan/ccan/io/poll.c:449
    #16 0x564ee89b3c3b in io_loop_with_timers lightningd/io_loop_with_timers.c:24
    #17 0x564ee89ba540 in main lightningd/lightningd.c:822
    #18 0x7f4dc2143b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell added a commit that referenced this issue Jun 30, 2019
Direct leak of 32 byte(s) in 1 object(s) allocated from:
    #0 0x7f7678ee863e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x55f8c7b0fce5 in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x55f8c7b1064f in double_table ccan/ccan/htable/htable.c:226
    #3 0x55f8c7b10b19 in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x55f8c7afac63 in scriptpubkeyset_add wallet/txfilter.c:30
    #5 0x55f8c7afafce in txfilter_add_scriptpubkey wallet/txfilter.c:77
    #6 0x55f8c7afb05f in txfilter_add_derkey wallet/txfilter.c:91
    #7 0x55f8c7aa4d67 in init_txfilter lightningd/lightningd.c:482
    #8 0x55f8c7aa52d8 in main lightningd/lightningd.c:721
    #9 0x7f767889ab6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7f05f389563e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x55cac1e6bc99 in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x55cac1e6c603 in double_table ccan/ccan/htable/htable.c:226
    #3 0x55cac1e6cacd in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x55cac1e56e48 in outpointset_add wallet/txfilter.c:61
    #5 0x55cac1e57162 in outpointfilter_add wallet/txfilter.c:116
    #6 0x55cac1e5ea3a in wallet_utxoset_add wallet/wallet.c:2365
    #7 0x55cac1deddc2 in topo_add_utxos lightningd/chaintopology.c:603
    #8 0x55cac1dedeac in add_tip lightningd/chaintopology.c:620
    #9 0x55cac1dee2de in have_new_block lightningd/chaintopology.c:694
    #10 0x55cac1deaab0 in process_rawblock lightningd/bitcoind.c:466
    #11 0x55cac1de9fb4 in bcli_finished lightningd/bitcoind.c:214
    #12 0x55cac1e6f5be in destroy_conn ccan/ccan/io/poll.c:244
    #13 0x55cac1e6f5de in destroy_conn_close_fd ccan/ccan/io/poll.c:250
    #14 0x55cac1e7baf5 in notify ccan/ccan/tal/tal.c:235
    #15 0x55cac1e7bfe4 in del_tree ccan/ccan/tal/tal.c:397
    #16 0x55cac1e7c370 in tal_free ccan/ccan/tal/tal.c:481
    #17 0x55cac1e6dddd in io_close ccan/ccan/io/io.c:450
    #18 0x55cac1e6fcf9 in io_loop ccan/ccan/io/poll.c:449
    #19 0x55cac1dfac66 in io_loop_with_timers lightningd/io_loop_with_timers.c:24
    #20 0x55cac1e0156b in main lightningd/lightningd.c:822
    #21 0x7f05f3247b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell added a commit that referenced this issue Jun 30, 2019
Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7ff02889063e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x555ce2ad8d2e in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x555ce2ad9698 in double_table ccan/ccan/htable/htable.c:226
    #3 0x555ce2ad9b62 in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x555ce2a638e4 in htlc_in_map_add lightningd/htlc_end.h:113
    #5 0x555ce2a63beb in connect_htlc_in lightningd/htlc_end.c:39
    #6 0x555ce2a85cbc in channel_added_their_htlc lightningd/peer_htlcs.c:1382
    #7 0x555ce2a860e1 in peer_got_commitsig lightningd/peer_htlcs.c:1466
    #8 0x555ce2a5db04 in channel_msg lightningd/channel_control.c:228
    #9 0x555ce2a8d393 in sd_msg_read lightningd/subd.c:474
    #10 0x555ce2ada157 in next_plan ccan/ccan/io/io.c:59
    #11 0x555ce2adacd4 in do_plan ccan/ccan/io/io.c:407
    #12 0x555ce2adad12 in io_ready ccan/ccan/io/io.c:417
    #13 0x555ce2adcd67 in io_loop ccan/ccan/io/poll.c:445
    #14 0x555ce2a67c66 in io_loop_with_timers lightningd/io_loop_with_timers.c:24
    #15 0x555ce2a6e56b in main lightningd/lightningd.c:822
    #16 0x7ff028242b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
ulrichard pushed a commit to ulrichard/lightning that referenced this issue May 20, 2021
Changed "warmstart" to "coldstart" (inverted semantic).

Modified all satoshi variable names to have "_sat" suffixes.

Removed deprecated Phase 1 fields.

Added push_value to hsm_ready_channel.  Added sanity checking to hsm_ready_channel stub.

Fixed issues w/ rebase onto upstream/master.

Replaced tx->input_amounts and tx->output_witscripts with tx->psbt.

Fixed redeemscript confusion. (ElementsProject#11)

Fix remote-hsmd CI (ElementsProject#12)

* adding g++, protobuf, grpc++
* add libgrpc++-dev to .travis.yml
* wget remotesigner.proto
* added libprotobuf-dev
* fixed compilation warnings/errors
* use Get instead of [] in protobuf seq
* fixed include sort order
* added protobuf generated files to skip nogen checks
* fixed gensrc pattern
* added libstdc++6
* fixed python spacing
* added specified inclusion guards
* renamed c++ header extensions to hpp to avoid c checks
* guard satoshi amount calculations
* libstdc++-6-dev
* added `ln -s /usr/lib/x86_64-linux-gnu/libstdc++.so.6 /usr/lib/x86_64-linux-gnu/libstdc++.so`

Enable handle_sign_withdrawal_tx signing of PSBT. (ElementsProject#13)

* Converted debug logging to json format.
* Switched status_debug to STATUS_DEBUG macro.

Skip two integration tests which mutate channel_nonce.

fixed nit w/ run-one-test

Added NodeConfig::NATIVE to InitReq. (ElementsProject#14)

Converted remotesigner.PubKey from 64 byte raw to 33 byte compressed DER. (ElementsProject#15)

Updated to channel_nonce0. (ElementsProject#17)

Replaced option_static_remotekey w/ CommitmentType. (ElementsProject#18)

* Replaced option_static_remotekey w/ CommitmentType.

* Improved NOTES

Upgraded SignFundingTxRequest KeyIndex to KeyPath. (ElementsProject#19)

Adapted to CloseInfo moving into KeyLocator

TxOut was replaced in API w/ it's only field.

Adapted to remotesigner api-doc mods.

Set input to 0 explicitly.

SignCounterpartyHtlcToUs -> SignCounterpartyHtlcSweep SignJusticeTxToUs -> SignJusticeSweep SignDelayedPaymentToUs -> SignDelayedSweep (ElementsProject#20)
vincenzopalazzo added a commit to vincenzopalazzo/lightning that referenced this issue Feb 13, 2023
This will fix a crash that I caused on armv7
and by looking inside the coredump with gdb
(by adding an assert on n that must be
different from null) I get the following stacktrace

```
(gdb) bt
\#0  0x00000000 in ?? ()
\#1  0x0043a038 in send_backtrace (why=0xbe9e3600 "FATAL SIGNAL 11") at common/daemon.c:36
\#2  0x0043a0ec in crashdump (sig=11) at common/daemon.c:46
\#3  <signal handler called>
\#4  0x00406d04 in node_announcement (map=0x938ecc, nann_off=495146) at common/gossmap.c:586
\#5  0x00406fec in map_catchup (map=0x938ecc, num_rejected=0xbe9e3a40) at common/gossmap.c:643
\#6  0x004073a4 in load_gossip_store (map=0x938ecc, num_rejected=0xbe9e3a40) at common/gossmap.c:697
\ElementsProject#7  0x00408244 in gossmap_load (ctx=0x0, filename=0x4e16b8 "gossip_store", num_channel_updates_rejected=0xbe9e3a40) at common/gossmap.c:976
\ElementsProject#8  0x0041a548 in init (p=0x93831c, buf=0x9399d4 "\n\n{\"jsonrpc\":\"2.0\",\"id\":\"cln:init#25\",\"method\":\"init\",\"params\":{\"options\":{},\"configuration\":{\"lightning-dir\":\"/home/vincent/.lightning/testnet\",\"rpc-file\":\"lightning-rpc\",\"startup\":true,\"network\":\"te"..., config=0x939cdc) at plugins/topology.c:622
\ElementsProject#9  0x0041e5d0 in handle_init (cmd=0x938934, buf=0x9399d4 "\n\n{\"jsonrpc\":\"2.0\",\"id\":\"cln:init#25\",\"method\":\"init\",\"params\":{\"options\":{},\"configuration\":{\"lightning-dir\":\"/home/vincent/.lightning/testnet\",\"rpc-file\":\"lightning-rpc\",\"startup\":true,\"network\":\"te"..., params=0x939c8c)
    at plugins/libplugin.c:1208
\ElementsProject#10 0x0041fc04 in ld_command_handle (plugin=0x93831c, toks=0x939bec) at plugins/libplugin.c:1572
\ElementsProject#11 0x00420050 in ld_read_json_one (plugin=0x93831c) at plugins/libplugin.c:1667
\ElementsProject#12 0x004201bc in ld_read_json (conn=0x9391c4, plugin=0x93831c) at plugins/libplugin.c:1687
\ElementsProject#13 0x004cb82c in next_plan (conn=0x9391c4, plan=0x9391d8) at ccan/ccan/io/io.c:59
\ElementsProject#14 0x004cc67c in do_plan (conn=0x9391c4, plan=0x9391d8, idle_on_epipe=false) at ccan/ccan/io/io.c:407
\ElementsProject#15 0x004cc6dc in io_ready (conn=0x9391c4, pollflags=1) at ccan/ccan/io/io.c:417
\ElementsProject#16 0x004cf8cc in io_loop (timers=0x9383c4, expired=0xbe9e3ce4) at ccan/ccan/io/poll.c:453
\ElementsProject#17 0x00420af4 in plugin_main (argv=0xbe9e3eb4, init=0x41a46c <init>, restartability=PLUGIN_STATIC, init_rpc=true, features=0x0, commands=0x6167e8 <commands>, num_commands=4, notif_subs=0x0, num_notif_subs=0, hook_subs=0x0, num_hook_subs=0, notif_topics=0x0, num_notif_topics=0) at plugins/libplugin.c:1891
\ElementsProject#18 0x0041a6f8 in main (argc=1, argv=0xbe9e3eb4) at plugins/topology.c:679
```

I do not know if this is a solution because I do not know
when I can parse a node announcement for a node that
it is not longer in the gossip map.

So, I hope this is just usefult for @rustyrussell

Changelog-Fixes: fixes `FATAL SIGNAL 11` on gossmap node announcement parsing.

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
vincenzopalazzo added a commit to vincenzopalazzo/lightning that referenced this issue Feb 13, 2023
This will fix a crash that I caused on armv7
and by looking inside the coredump with gdb
(by adding an assert on n that must be
different from null) I get the following stacktrace

```
(gdb) bt
\#0  0x00000000 in ?? ()
\#1  0x0043a038 in send_backtrace (why=0xbe9e3600 "FATAL SIGNAL 11") at common/daemon.c:36
\#2  0x0043a0ec in crashdump (sig=11) at common/daemon.c:46
\#3  <signal handler called>
\#4  0x00406d04 in node_announcement (map=0x938ecc, nann_off=495146) at common/gossmap.c:586
\#5  0x00406fec in map_catchup (map=0x938ecc, num_rejected=0xbe9e3a40) at common/gossmap.c:643
\#6  0x004073a4 in load_gossip_store (map=0x938ecc, num_rejected=0xbe9e3a40) at common/gossmap.c:697
\ElementsProject#7  0x00408244 in gossmap_load (ctx=0x0, filename=0x4e16b8 "gossip_store", num_channel_updates_rejected=0xbe9e3a40) at common/gossmap.c:976
\ElementsProject#8  0x0041a548 in init (p=0x93831c, buf=0x9399d4 "\n\n{\"jsonrpc\":\"2.0\",\"id\":\"cln:init#25\",\"method\":\"init\",\"params\":{\"options\":{},\"configuration\":{\"lightning-dir\":\"/home/vincent/.lightning/testnet\",\"rpc-file\":\"lightning-rpc\",\"startup\":true,\"network\":\"te"..., config=0x939cdc) at plugins/topology.c:622
\ElementsProject#9  0x0041e5d0 in handle_init (cmd=0x938934, buf=0x9399d4 "\n\n{\"jsonrpc\":\"2.0\",\"id\":\"cln:init#25\",\"method\":\"init\",\"params\":{\"options\":{},\"configuration\":{\"lightning-dir\":\"/home/vincent/.lightning/testnet\",\"rpc-file\":\"lightning-rpc\",\"startup\":true,\"network\":\"te"..., params=0x939c8c)
    at plugins/libplugin.c:1208
\ElementsProject#10 0x0041fc04 in ld_command_handle (plugin=0x93831c, toks=0x939bec) at plugins/libplugin.c:1572
\ElementsProject#11 0x00420050 in ld_read_json_one (plugin=0x93831c) at plugins/libplugin.c:1667
\ElementsProject#12 0x004201bc in ld_read_json (conn=0x9391c4, plugin=0x93831c) at plugins/libplugin.c:1687
\ElementsProject#13 0x004cb82c in next_plan (conn=0x9391c4, plan=0x9391d8) at ccan/ccan/io/io.c:59
\ElementsProject#14 0x004cc67c in do_plan (conn=0x9391c4, plan=0x9391d8, idle_on_epipe=false) at ccan/ccan/io/io.c:407
\ElementsProject#15 0x004cc6dc in io_ready (conn=0x9391c4, pollflags=1) at ccan/ccan/io/io.c:417
\ElementsProject#16 0x004cf8cc in io_loop (timers=0x9383c4, expired=0xbe9e3ce4) at ccan/ccan/io/poll.c:453
\ElementsProject#17 0x00420af4 in plugin_main (argv=0xbe9e3eb4, init=0x41a46c <init>, restartability=PLUGIN_STATIC, init_rpc=true, features=0x0, commands=0x6167e8 <commands>, num_commands=4, notif_subs=0x0, num_notif_subs=0, hook_subs=0x0, num_hook_subs=0, notif_topics=0x0, num_notif_topics=0) at plugins/libplugin.c:1891
\ElementsProject#18 0x0041a6f8 in main (argc=1, argv=0xbe9e3eb4) at plugins/topology.c:679
```

I do not know if this is a solution because I do not know
when I can parse a node announcement for a node that
it is not longer in the gossip map.

So, I hope this is just usefult for @rustyrussell

Changelog-Fix: fixes `FATAL SIGNAL 11` on gossmap node announcement parsing.

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
vincenzopalazzo added a commit to vincenzopalazzo/lightning that referenced this issue Feb 13, 2023
This will fix a crash that I caused on armv7
and by looking inside the coredump with gdb
(by adding an assert on n that must be
different from null) I get the following stacktrace

```
(gdb) bt
\#0  0x00000000 in ?? ()
\#1  0x0043a038 in send_backtrace (why=0xbe9e3600 "FATAL SIGNAL 11") at common/daemon.c:36
\#2  0x0043a0ec in crashdump (sig=11) at common/daemon.c:46
\#3  <signal handler called>
\#4  0x00406d04 in node_announcement (map=0x938ecc, nann_off=495146) at common/gossmap.c:586
\#5  0x00406fec in map_catchup (map=0x938ecc, num_rejected=0xbe9e3a40) at common/gossmap.c:643
\#6  0x004073a4 in load_gossip_store (map=0x938ecc, num_rejected=0xbe9e3a40) at common/gossmap.c:697
\ElementsProject#7  0x00408244 in gossmap_load (ctx=0x0, filename=0x4e16b8 "gossip_store", num_channel_updates_rejected=0xbe9e3a40) at common/gossmap.c:976
\ElementsProject#8  0x0041a548 in init (p=0x93831c, buf=0x9399d4 "\n\n{\"jsonrpc\":\"2.0\",\"id\":\"cln:init#25\",\"method\":\"init\",\"params\":{\"options\":{},\"configuration\":{\"lightning-dir\":\"/home/vincent/.lightning/testnet\",\"rpc-file\":\"lightning-rpc\",\"startup\":true,\"network\":\"te"..., config=0x939cdc) at plugins/topology.c:622
\ElementsProject#9  0x0041e5d0 in handle_init (cmd=0x938934, buf=0x9399d4 "\n\n{\"jsonrpc\":\"2.0\",\"id\":\"cln:init#25\",\"method\":\"init\",\"params\":{\"options\":{},\"configuration\":{\"lightning-dir\":\"/home/vincent/.lightning/testnet\",\"rpc-file\":\"lightning-rpc\",\"startup\":true,\"network\":\"te"..., params=0x939c8c)
    at plugins/libplugin.c:1208
\ElementsProject#10 0x0041fc04 in ld_command_handle (plugin=0x93831c, toks=0x939bec) at plugins/libplugin.c:1572
\ElementsProject#11 0x00420050 in ld_read_json_one (plugin=0x93831c) at plugins/libplugin.c:1667
\ElementsProject#12 0x004201bc in ld_read_json (conn=0x9391c4, plugin=0x93831c) at plugins/libplugin.c:1687
\ElementsProject#13 0x004cb82c in next_plan (conn=0x9391c4, plan=0x9391d8) at ccan/ccan/io/io.c:59
\ElementsProject#14 0x004cc67c in do_plan (conn=0x9391c4, plan=0x9391d8, idle_on_epipe=false) at ccan/ccan/io/io.c:407
\ElementsProject#15 0x004cc6dc in io_ready (conn=0x9391c4, pollflags=1) at ccan/ccan/io/io.c:417
\ElementsProject#16 0x004cf8cc in io_loop (timers=0x9383c4, expired=0xbe9e3ce4) at ccan/ccan/io/poll.c:453
\ElementsProject#17 0x00420af4 in plugin_main (argv=0xbe9e3eb4, init=0x41a46c <init>, restartability=PLUGIN_STATIC, init_rpc=true, features=0x0, commands=0x6167e8 <commands>, num_commands=4, notif_subs=0x0, num_notif_subs=0, hook_subs=0x0, num_hook_subs=0, notif_topics=0x0, num_notif_topics=0) at plugins/libplugin.c:1891
\ElementsProject#18 0x0041a6f8 in main (argc=1, argv=0xbe9e3eb4) at plugins/topology.c:679
```

I do not know if this is a solution because I do not know
when I can parse a node announcement for a node that
it is not longer in the gossip map.

So, I hope this is just usefult for @rustyrussell

Changelog-Fixed: fixes `FATAL SIGNAL 11` on gossmap node announcement parsing.

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
endothermicdev pushed a commit that referenced this issue Feb 13, 2023
This will fix a crash that I caused on armv7
and by looking inside the coredump with gdb
(by adding an assert on n that must be
different from null) I get the following stacktrace

```
(gdb) bt
\#0  0x00000000 in ?? ()
\#1  0x0043a038 in send_backtrace (why=0xbe9e3600 "FATAL SIGNAL 11") at common/daemon.c:36
\#2  0x0043a0ec in crashdump (sig=11) at common/daemon.c:46
\#3  <signal handler called>
\#4  0x00406d04 in node_announcement (map=0x938ecc, nann_off=495146) at common/gossmap.c:586
\#5  0x00406fec in map_catchup (map=0x938ecc, num_rejected=0xbe9e3a40) at common/gossmap.c:643
\#6  0x004073a4 in load_gossip_store (map=0x938ecc, num_rejected=0xbe9e3a40) at common/gossmap.c:697
\#7  0x00408244 in gossmap_load (ctx=0x0, filename=0x4e16b8 "gossip_store", num_channel_updates_rejected=0xbe9e3a40) at common/gossmap.c:976
\#8  0x0041a548 in init (p=0x93831c, buf=0x9399d4 "\n\n{\"jsonrpc\":\"2.0\",\"id\":\"cln:init#25\",\"method\":\"init\",\"params\":{\"options\":{},\"configuration\":{\"lightning-dir\":\"/home/vincent/.lightning/testnet\",\"rpc-file\":\"lightning-rpc\",\"startup\":true,\"network\":\"te"..., config=0x939cdc) at plugins/topology.c:622
\#9  0x0041e5d0 in handle_init (cmd=0x938934, buf=0x9399d4 "\n\n{\"jsonrpc\":\"2.0\",\"id\":\"cln:init#25\",\"method\":\"init\",\"params\":{\"options\":{},\"configuration\":{\"lightning-dir\":\"/home/vincent/.lightning/testnet\",\"rpc-file\":\"lightning-rpc\",\"startup\":true,\"network\":\"te"..., params=0x939c8c)
    at plugins/libplugin.c:1208
\#10 0x0041fc04 in ld_command_handle (plugin=0x93831c, toks=0x939bec) at plugins/libplugin.c:1572
\#11 0x00420050 in ld_read_json_one (plugin=0x93831c) at plugins/libplugin.c:1667
\#12 0x004201bc in ld_read_json (conn=0x9391c4, plugin=0x93831c) at plugins/libplugin.c:1687
\#13 0x004cb82c in next_plan (conn=0x9391c4, plan=0x9391d8) at ccan/ccan/io/io.c:59
\#14 0x004cc67c in do_plan (conn=0x9391c4, plan=0x9391d8, idle_on_epipe=false) at ccan/ccan/io/io.c:407
\#15 0x004cc6dc in io_ready (conn=0x9391c4, pollflags=1) at ccan/ccan/io/io.c:417
\#16 0x004cf8cc in io_loop (timers=0x9383c4, expired=0xbe9e3ce4) at ccan/ccan/io/poll.c:453
\#17 0x00420af4 in plugin_main (argv=0xbe9e3eb4, init=0x41a46c <init>, restartability=PLUGIN_STATIC, init_rpc=true, features=0x0, commands=0x6167e8 <commands>, num_commands=4, notif_subs=0x0, num_notif_subs=0, hook_subs=0x0, num_hook_subs=0, notif_topics=0x0, num_notif_topics=0) at plugins/libplugin.c:1891
\#18 0x0041a6f8 in main (argc=1, argv=0xbe9e3eb4) at plugins/topology.c:679
```

I do not know if this is a solution because I do not know
when I can parse a node announcement for a node that
it is not longer in the gossip map.

So, I hope this is just usefult for @rustyrussell

Changelog-Fixed: fixes `FATAL SIGNAL 11` on gossmap node announcement parsing.

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
ddustin pushed a commit to ddustin/lightning that referenced this issue Apr 11, 2023
This will fix a crash that I caused on armv7
and by looking inside the coredump with gdb
(by adding an assert on n that must be
different from null) I get the following stacktrace

```
(gdb) bt
\#0  0x00000000 in ?? ()
\#1  0x0043a038 in send_backtrace (why=0xbe9e3600 "FATAL SIGNAL 11") at common/daemon.c:36
\#2  0x0043a0ec in crashdump (sig=11) at common/daemon.c:46
\#3  <signal handler called>
\#4  0x00406d04 in node_announcement (map=0x938ecc, nann_off=495146) at common/gossmap.c:586
\#5  0x00406fec in map_catchup (map=0x938ecc, num_rejected=0xbe9e3a40) at common/gossmap.c:643
\ElementsProject#6  0x004073a4 in load_gossip_store (map=0x938ecc, num_rejected=0xbe9e3a40) at common/gossmap.c:697
\ElementsProject#7  0x00408244 in gossmap_load (ctx=0x0, filename=0x4e16b8 "gossip_store", num_channel_updates_rejected=0xbe9e3a40) at common/gossmap.c:976
\ElementsProject#8  0x0041a548 in init (p=0x93831c, buf=0x9399d4 "\n\n{\"jsonrpc\":\"2.0\",\"id\":\"cln:init#25\",\"method\":\"init\",\"params\":{\"options\":{},\"configuration\":{\"lightning-dir\":\"/home/vincent/.lightning/testnet\",\"rpc-file\":\"lightning-rpc\",\"startup\":true,\"network\":\"te"..., config=0x939cdc) at plugins/topology.c:622
\ElementsProject#9  0x0041e5d0 in handle_init (cmd=0x938934, buf=0x9399d4 "\n\n{\"jsonrpc\":\"2.0\",\"id\":\"cln:init#25\",\"method\":\"init\",\"params\":{\"options\":{},\"configuration\":{\"lightning-dir\":\"/home/vincent/.lightning/testnet\",\"rpc-file\":\"lightning-rpc\",\"startup\":true,\"network\":\"te"..., params=0x939c8c)
    at plugins/libplugin.c:1208
\ElementsProject#10 0x0041fc04 in ld_command_handle (plugin=0x93831c, toks=0x939bec) at plugins/libplugin.c:1572
\ElementsProject#11 0x00420050 in ld_read_json_one (plugin=0x93831c) at plugins/libplugin.c:1667
\ElementsProject#12 0x004201bc in ld_read_json (conn=0x9391c4, plugin=0x93831c) at plugins/libplugin.c:1687
\ElementsProject#13 0x004cb82c in next_plan (conn=0x9391c4, plan=0x9391d8) at ccan/ccan/io/io.c:59
\ElementsProject#14 0x004cc67c in do_plan (conn=0x9391c4, plan=0x9391d8, idle_on_epipe=false) at ccan/ccan/io/io.c:407
\ElementsProject#15 0x004cc6dc in io_ready (conn=0x9391c4, pollflags=1) at ccan/ccan/io/io.c:417
\ElementsProject#16 0x004cf8cc in io_loop (timers=0x9383c4, expired=0xbe9e3ce4) at ccan/ccan/io/poll.c:453
\ElementsProject#17 0x00420af4 in plugin_main (argv=0xbe9e3eb4, init=0x41a46c <init>, restartability=PLUGIN_STATIC, init_rpc=true, features=0x0, commands=0x6167e8 <commands>, num_commands=4, notif_subs=0x0, num_notif_subs=0, hook_subs=0x0, num_hook_subs=0, notif_topics=0x0, num_notif_topics=0) at plugins/libplugin.c:1891
\ElementsProject#18 0x0041a6f8 in main (argc=1, argv=0xbe9e3eb4) at plugins/topology.c:679
```

I do not know if this is a solution because I do not know
when I can parse a node announcement for a node that
it is not longer in the gossip map.

So, I hope this is just usefult for @rustyrussell

Changelog-Fixed: fixes `FATAL SIGNAL 11` on gossmap node announcement parsing.

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
gkrizek pushed a commit to voltagecloud/lightning that referenced this issue Apr 26, 2023
This will fix a crash that I caused on armv7
and by looking inside the coredump with gdb
(by adding an assert on n that must be
different from null) I get the following stacktrace

```
(gdb) bt
\#0  0x00000000 in ?? ()
\#1  0x0043a038 in send_backtrace (why=0xbe9e3600 "FATAL SIGNAL 11") at common/daemon.c:36
\#2  0x0043a0ec in crashdump (sig=11) at common/daemon.c:46
\#3  <signal handler called>
\ElementsProject#4  0x00406d04 in node_announcement (map=0x938ecc, nann_off=495146) at common/gossmap.c:586
\ElementsProject#5  0x00406fec in map_catchup (map=0x938ecc, num_rejected=0xbe9e3a40) at common/gossmap.c:643
\ElementsProject#6  0x004073a4 in load_gossip_store (map=0x938ecc, num_rejected=0xbe9e3a40) at common/gossmap.c:697
\ElementsProject#7  0x00408244 in gossmap_load (ctx=0x0, filename=0x4e16b8 "gossip_store", num_channel_updates_rejected=0xbe9e3a40) at common/gossmap.c:976
\ElementsProject#8  0x0041a548 in init (p=0x93831c, buf=0x9399d4 "\n\n{\"jsonrpc\":\"2.0\",\"id\":\"cln:init#25\",\"method\":\"init\",\"params\":{\"options\":{},\"configuration\":{\"lightning-dir\":\"/home/vincent/.lightning/testnet\",\"rpc-file\":\"lightning-rpc\",\"startup\":true,\"network\":\"te"..., config=0x939cdc) at plugins/topology.c:622
\ElementsProject#9  0x0041e5d0 in handle_init (cmd=0x938934, buf=0x9399d4 "\n\n{\"jsonrpc\":\"2.0\",\"id\":\"cln:init#25\",\"method\":\"init\",\"params\":{\"options\":{},\"configuration\":{\"lightning-dir\":\"/home/vincent/.lightning/testnet\",\"rpc-file\":\"lightning-rpc\",\"startup\":true,\"network\":\"te"..., params=0x939c8c)
    at plugins/libplugin.c:1208
\ElementsProject#10 0x0041fc04 in ld_command_handle (plugin=0x93831c, toks=0x939bec) at plugins/libplugin.c:1572
\ElementsProject#11 0x00420050 in ld_read_json_one (plugin=0x93831c) at plugins/libplugin.c:1667
\ElementsProject#12 0x004201bc in ld_read_json (conn=0x9391c4, plugin=0x93831c) at plugins/libplugin.c:1687
\ElementsProject#13 0x004cb82c in next_plan (conn=0x9391c4, plan=0x9391d8) at ccan/ccan/io/io.c:59
\ElementsProject#14 0x004cc67c in do_plan (conn=0x9391c4, plan=0x9391d8, idle_on_epipe=false) at ccan/ccan/io/io.c:407
\ElementsProject#15 0x004cc6dc in io_ready (conn=0x9391c4, pollflags=1) at ccan/ccan/io/io.c:417
\ElementsProject#16 0x004cf8cc in io_loop (timers=0x9383c4, expired=0xbe9e3ce4) at ccan/ccan/io/poll.c:453
\ElementsProject#17 0x00420af4 in plugin_main (argv=0xbe9e3eb4, init=0x41a46c <init>, restartability=PLUGIN_STATIC, init_rpc=true, features=0x0, commands=0x6167e8 <commands>, num_commands=4, notif_subs=0x0, num_notif_subs=0, hook_subs=0x0, num_hook_subs=0, notif_topics=0x0, num_notif_topics=0) at plugins/libplugin.c:1891
\ElementsProject#18 0x0041a6f8 in main (argc=1, argv=0xbe9e3eb4) at plugins/topology.c:679
```

I do not know if this is a solution because I do not know
when I can parse a node announcement for a node that
it is not longer in the gossip map.

So, I hope this is just usefult for @rustyrussell

Changelog-Fixed: fixes `FATAL SIGNAL 11` on gossmap node announcement parsing.

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
ddustin pushed a commit to ddustin/lightning that referenced this issue May 12, 2023
This will fix a crash that I caused on armv7
and by looking inside the coredump with gdb
(by adding an assert on n that must be
different from null) I get the following stacktrace

```
(gdb) bt
\#0  0x00000000 in ?? ()
\#1  0x0043a038 in send_backtrace (why=0xbe9e3600 "FATAL SIGNAL 11") at common/daemon.c:36
\#2  0x0043a0ec in crashdump (sig=11) at common/daemon.c:46
\#3  <signal handler called>
\#4  0x00406d04 in node_announcement (map=0x938ecc, nann_off=495146) at common/gossmap.c:586
\#5  0x00406fec in map_catchup (map=0x938ecc, num_rejected=0xbe9e3a40) at common/gossmap.c:643
\ElementsProject#6  0x004073a4 in load_gossip_store (map=0x938ecc, num_rejected=0xbe9e3a40) at common/gossmap.c:697
\ElementsProject#7  0x00408244 in gossmap_load (ctx=0x0, filename=0x4e16b8 "gossip_store", num_channel_updates_rejected=0xbe9e3a40) at common/gossmap.c:976
\ElementsProject#8  0x0041a548 in init (p=0x93831c, buf=0x9399d4 "\n\n{\"jsonrpc\":\"2.0\",\"id\":\"cln:init#25\",\"method\":\"init\",\"params\":{\"options\":{},\"configuration\":{\"lightning-dir\":\"/home/vincent/.lightning/testnet\",\"rpc-file\":\"lightning-rpc\",\"startup\":true,\"network\":\"te"..., config=0x939cdc) at plugins/topology.c:622
\ElementsProject#9  0x0041e5d0 in handle_init (cmd=0x938934, buf=0x9399d4 "\n\n{\"jsonrpc\":\"2.0\",\"id\":\"cln:init#25\",\"method\":\"init\",\"params\":{\"options\":{},\"configuration\":{\"lightning-dir\":\"/home/vincent/.lightning/testnet\",\"rpc-file\":\"lightning-rpc\",\"startup\":true,\"network\":\"te"..., params=0x939c8c)
    at plugins/libplugin.c:1208
\ElementsProject#10 0x0041fc04 in ld_command_handle (plugin=0x93831c, toks=0x939bec) at plugins/libplugin.c:1572
\ElementsProject#11 0x00420050 in ld_read_json_one (plugin=0x93831c) at plugins/libplugin.c:1667
\ElementsProject#12 0x004201bc in ld_read_json (conn=0x9391c4, plugin=0x93831c) at plugins/libplugin.c:1687
\ElementsProject#13 0x004cb82c in next_plan (conn=0x9391c4, plan=0x9391d8) at ccan/ccan/io/io.c:59
\ElementsProject#14 0x004cc67c in do_plan (conn=0x9391c4, plan=0x9391d8, idle_on_epipe=false) at ccan/ccan/io/io.c:407
\ElementsProject#15 0x004cc6dc in io_ready (conn=0x9391c4, pollflags=1) at ccan/ccan/io/io.c:417
\ElementsProject#16 0x004cf8cc in io_loop (timers=0x9383c4, expired=0xbe9e3ce4) at ccan/ccan/io/poll.c:453
\ElementsProject#17 0x00420af4 in plugin_main (argv=0xbe9e3eb4, init=0x41a46c <init>, restartability=PLUGIN_STATIC, init_rpc=true, features=0x0, commands=0x6167e8 <commands>, num_commands=4, notif_subs=0x0, num_notif_subs=0, hook_subs=0x0, num_hook_subs=0, notif_topics=0x0, num_notif_topics=0) at plugins/libplugin.c:1891
\ElementsProject#18 0x0041a6f8 in main (argc=1, argv=0xbe9e3eb4) at plugins/topology.c:679
```

I do not know if this is a solution because I do not know
when I can parse a node announcement for a node that
it is not longer in the gossip map.

So, I hope this is just usefult for @rustyrussell

Changelog-Fixed: fixes `FATAL SIGNAL 11` on gossmap node announcement parsing.

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants