DB changes to shrink unneeded HTLCs, only pass minimal set to onchaind. #4850
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rustyrussell
force-pushed
the
guilt/smaller-db
branch
7 times, most recently
from
378c9e8
to
9c67881
Compare
sendonionmessage can fail when sending a reply, either because the reply had a bad first peer, or because it went offline. The latter happens in CI, which is how I found this. Also fixed typo "onio" -> "onion". Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
How did this pass CI? I saw this break in my PR, but it's (long) broken in master. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We are usually dealing with unsigned values, so use this. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
All enums in the db should be wrapped this way on reading/writing them. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And initialize max to current height max when htlcs are already dead. Turns out (thanks CI!) that MAX() of multiple columns is GREATEST() in Postgres. That's clearer (MAX is used elsewhere for single columns), so translate on the sqlite3 side. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In particular, the onion and errors can be large, but now we'll never need to retransmit them. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is particularly useful after our recent field deletion: before: 362,573,824 bytes after: 124,190,720 bytes Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> Changelog-Changed: db: removal of old HTLC information and vacuuming shrinks large lightningd.sqlite3 by a factor of 2-3.
This makes more sense than two args. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I started pulling this thread, and the entire codebase got unravelled. Oh well, it's done now! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This makes init a two-stage, and causes some code hoisting. And we can now send all the HTLCs in a single message, since we have an 128MB limit and each HTLC is 37 bytes. This breaks the onchaind stresstest, which uses canned internal messages. It's time to finally delete that. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In particular, we could get depth notifications. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This will make closing long-lived channels more efficient, and it's just nicer. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
``` [gw1] [ 98%] PASSED tests/test_wallet.py::test_hsmtool_dump_descriptors tests/test_wallet.py::test_fundchannel_listtransaction [gw0] [ 98%] PASSED tests/test_plugin.py::test_channel_opened_notification tests/test_wallet.py::test_hsmtool_generatehsm [gw0] [ 98%] PASSED tests/test_wallet.py::test_hsmtool_generatehsm tests/test_wallet.py::test_withdraw_nlocktime_fuzz [gw1] [ 98%] ERROR tests/test_wallet.py::test_fundchannel_listtransaction tests/test_wallet.py::test_fundchannel_listtransaction tests/test_wallet.py::test_withdraw_nlocktime_fuzz tests/test_wallet.py::test_fundchannel_listtransaction [gw0] [ 99%] ERROR tests/test_wallet.py::test_withdraw_nlocktime_fuzz tests/test_wallet.py::test_multiwithdraw_simple [gw1] [ 99%] ERROR tests/test_wallet.py::test_fundchannel_listtransaction tests/test_wallet.py::test_withdraw_nlocktime tests/test_wallet.py::test_multiwithdraw_simple tests/test_wallet.py::test_withdraw_nlocktime tests/test_wallet.py::test_multiwithdraw_simple tests/test_wallet.py::test_withdraw_nlocktime [gw0] [ 99%] ERROR tests/test_wallet.py::test_multiwithdraw_simple tests/test_wallet.py::test_repro_4258 [gw1] [ 99%] ERROR tests/test_wallet.py::test_withdraw_nlocktime ... 2021-10-12 06:36:09.203 UTC [224552] STATEMENT: SELECT version FROM version LIMIT 1 2021-10-12 06:36:09.566 UTC [224523] PANIC: could not write to file "pg_wal/xlogtemp.224523": No space left on device 2021-10-12 06:36:09.566 UTC [224523] STATEMENT: VACUUM FULL; Error vacuuming db: BEGIN command failed: PANIC: could not write to file "pg_wal/xlogtemp.224523": No space left on device server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request. ```
rustyrussell
force-pushed
the
guilt/smaller-db
branch
from
9c67881
to
c9d1b7f
Compare
cdecker
approved these changes
Oct 15, 2021
| { | ||
| assert(!amount_msat_eq(amount, AMOUNT_MSAT(7206000))); |
There was a problem hiding this comment.
Where did this random constant come from?
There was a problem hiding this comment.
@rustyrussell ^^ (I guess I have seen a similar number and it was related to some altcoin pre-mine :-D ROTFL
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This became a bigger opus than intended, sorry! But everyone's dbs should shrink as a result.
Edit: rebased on #4863