discovery+rpc: expose graph synced status within GetInfo #3355

wpaulino · 2019-07-29T22:06:02Z

One thing to note is that the current implementation will lead to GetNetworkInfo blocking if it's called while lnd is waiting for the backend to be considered synced. This is because we don't initialize lnd until that's done first. This can be addressed, but I've decided to leave it out unless we think it's something definitely worth addressing.

Roasbeef · 2019-07-30T00:00:20Z

One thing to note is that the current implementation will lead to GetNetworkInfo blocking if it's called while lnd is waiting for the backend to be considered synced.

This'll have an affect on tools such as lndmon, as they won't be able to plot the increase in chans known over time. Still need to dive into the diff, but can't we just return false in the scenario that would cause us to block?

discovery/sync_manager.go

Roasbeef · 2019-07-30T00:03:02Z

rpcserver.go

@@ -4029,6 +4029,7 @@ func (r *rpcServer) GetNetworkInfo(ctx context.Context,
 		MaxChannelSize:       int64(maxChannelSize),
 		MedianChannelSizeSat: int64(medianChanSize),
 		NumZombieChans:       numZombies,
+		SyncedToGraph:        r.server.authGossiper.SyncManager().IsGraphSynced(),


If you make the return value channel, then we don't need to block here and can fall through to false in that case.

Went with a different approach instead.

cfromknecht · 2019-07-30T03:46:55Z

for some reason i was under the impression we wanted to expose a percentage rather than just a boolean, was that not the case? other the code changes look pretty good

wpaulino · 2019-07-30T04:36:17Z

@cfromknecht I went for the simplest approach possible as exposing a percentage seemed a bit more involved, though I'm not opposed to doing that.

cfromknecht

LGTM 👨‍🚀 We can defer on exposing a percentage-based indicator if it proves useful

sergioabril · 2019-07-31T23:49:50Z

@cfromknecht IMO, a percentage would be much more useful. The boolean is ok, but feels incomplete to me.

wpaulino · 2019-08-01T00:07:00Z

@sergioabril could you provide more insight for your use case of the percentage? Would it just be to expose the percentage to users in a higher level application? With the boolean, it should quickly flip to true unless it's your first time syncing the graph or you've been offline for a while.

sergioabril · 2019-08-01T00:41:57Z

Yes, exactly for that; but I'm thinking of mobile wallets, which can be offline for a few days/week, and where a more precise state description is needed for the user to wait. Desktop instances are probably fine with a boolean.

…

On 31 Jul 2019, at 17:07, Wilmer Paulino ***@***.***> wrote: @sergioabril could you provide more insight for your use case of the percentage? Would it just be to expose the percentage to users in a higher level application? With the boolean, it should quickly flip to true unless it's your first time syncing the graph or you've been offline for a while. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

cfromknecht · 2019-08-01T02:02:36Z

@wpaulino

With the boolean, it should quickly flip to true unless it's your first time syncing the graph or you've been offline for a while.

If we connect to a peer that doesn't respond, will this continue to show false indefinitely?

wpaulino · 2019-08-01T21:07:31Z

If we connect to a peer that doesn't respond, will this continue to show false indefinitely?

@cfromknecht wasn't the case previously, but should be fixed in the latest version!

cfromknecht · 2019-08-02T02:29:11Z

@cfromknecht wasn't the case previously, but should be fixed in the latest version!

just looked at the latest diff, how is this solved? if we send a query_channel_range and the initial peer we connect to is slow to reply, then is_synced will continue to show false. in the case of the app, that means a peer not responding can prevent you from making payments

wpaulino · 2019-08-02T02:32:53Z

See fcb6a8b. We’ll pick up on the status of the next historical sync if our initial one hasn’t completed by then. Did you have another solution in mind?

cfromknecht · 2019-08-02T02:46:13Z

See fcb6a8b. We’ll pick up on the status of the next historical sync if our initial one hasn’t completed by then.

i don't think we're talking about the same issue. the is_synced field can only become true if a peer responds to our query_channel_range message. if none of our peers respond, then it will always be false. if we gate payments in the app based on this field, then you will never be able to make a payment if you connect to poor/malicious peers.

Did you have another solution in mind?

one option is to add a deadline after which it will flip to true regardless, that way if a peer doesn't respond we're still covered.

in my mind, there's really three states:

UNKNOWN
SYNCING
SYNCED

when we come up, we start at UNKNOWN. after receiving any reply from our historical syncer[s] we then transition to:

SYNCING, if they have unknown channels, transitioning to SYNCED once a syncer completes
SYNCED, if they don't

my proposal is to add a timeout where if we stay in UNKNOWN for too long, we'll transition to SYNCED anyway and is_synced becomes true.

alternatively, these enums could be exposed over rpc to allow higher-level applications to act on. for example, if an app sees that we're still in UNKNOWN after some timeout, it can make the judgement call as to whether to allow payments or not

wpaulino · 2019-08-02T07:11:26Z

if we gate payments in the app based on this field, then you will never be able to make a payment if you connect to poor/malicious peers.

Why not just alert users rather than preventing them from completely making payments? Higher level applications can choose to interpret this field as they wish. They can decide to only gate payments up to some timeout as you suggested, gate payments regardless of how long it takes, alert users, or something else entirely.

when we come up, we start at UNKNOWN. after receiving any reply from our historical syncer[s] we then transition to:

SYNCING, if they have unknown channels, transitioning to SYNCED once a syncer completes

SYNCED, if they don't

my proposal is to add a timeout where if we stay in UNKNOWN for too long, we'll transition to > SYNCED anyway and is_synced becomes true.

Transitioning to true even though we haven't synced the graph seems to defeat the purpose of the flag, which is just to give an indication that we have done our part to sync the graph with at least one peer fully and will continue to do so through the consequent historical syncs. Just because they replied once doesn't mean we can guarantee they'll continue to do so, so we can still be in the SYNCING state. If they don't reply, do we flip back to UNKNOWN? Do we try another peer after a timeout? The additional complexity doesn't seem worth it.

alternatively, these enums could be exposed over rpc to allow higher-level applications to act on. for example, if an app sees that we're still in UNKNOWN after some timeout, it can make the judgement call as to whether to allow payments or not

They can already do this with how things are at the moment, i.e. synced_to_chain is false after some timeout, though I guess they wouldn't be able to differentiate between being stalled and actually syncing.

halseth · 2019-08-02T10:27:01Z

discovery/sync_manager.go

+				continue
+			}
+
+			// Otherwise, we'll track the peer we've performed a


interesting. So this is really an existing bug, but it wouldn't really matter that the initialHistoricalSyncCompleted was never set?

Exactly. It would matter as it could prevent us from transitioning to active syncers.

discovery/sync_manager.go

lnrpc/rpc.proto


        discovery: allow the SyncManager to report whether the graph is synced


        discovery: handle graph synced status after stalled initial historica…

…l sync This ensures that the graph synced status is marked true at some point once a historical sync has completed. Before this commit, a stalled historical sync could cause us to never mark the graph as synced.


        rpc: expose graph synced status within GetInfo

Roasbeef · 2019-08-07T02:33:02Z

On the question of if we need to care about a peer stalling: that seems to be more of an issue for our peer selection, and eclipse attack mitigations, no? If a peer isn't responding, then we can move to time them out, and rotate to another peer.

Our initial use case for this is also pretty simple, in just greying things out, or displaying this as an active indicator. Also practically, in the case of the application, they're connected to mostly known "golden" peers as well.

cfromknecht

LGTB 💰

Roasbeef

LGTM 🌋

githorray · 2019-08-13T20:57:30Z

I get "synced_to_graph": true via the getinfo rpc call, but not via lncli getinfo.

wpaulino · 2019-08-14T00:41:43Z

Thanks @githorray, fixed with #3400.

wpaulino requested a review from halseth as a code owner Jul 29, 2019

wpaulino added this to the 0.8.0 milestone Jul 29, 2019

Roasbeef added discovery enhancement v0.8.0 labels Jul 29, 2019

Roasbeef requested changes Jul 30, 2019

View changes

wpaulino force-pushed the wpaulino:is-graph-synced branch from 31661b2 to fec422b Jul 30, 2019

wpaulino requested a review from Roasbeef Jul 30, 2019

cfromknecht approved these changes Jul 31, 2019

View changes

wpaulino force-pushed the wpaulino:is-graph-synced branch from fec422b to 582d44b Jul 31, 2019

wpaulino force-pushed the wpaulino:is-graph-synced branch from 582d44b to 98348a4 Aug 1, 2019

wpaulino requested a review from cfromknecht Aug 1, 2019

wpaulino referenced this pull request Aug 1, 2019

discovery: prevent broadcast of anns received during initial graph sync #3366

Merged

halseth reviewed Aug 2, 2019

View changes

wpaulino requested a review from halseth Aug 6, 2019

cfromknecht reviewed Aug 6, 2019

View changes

discovery/sync_manager.go Show resolved Hide resolved

lnrpc/rpc.proto Outdated Show resolved Hide resolved

wpaulino added 2 commits Jul 27, 2019

discovery: allow the SyncManager to report whether the graph is synced

Verified

This commit was signed with a verified signature.

wpaulino Wilmer Paulino

GPG key ID: 6DF57B9F9514972F Learn about signing commits

af4234f

rpc: expose graph synced status within GetInfo

Verified

This commit was signed with a verified signature.

wpaulino Wilmer Paulino

GPG key ID: 6DF57B9F9514972F Learn about signing commits

Loading status checks…

8ca2c79

wpaulino force-pushed the wpaulino:is-graph-synced branch from 98348a4 to 8ca2c79 Aug 7, 2019

wpaulino requested a review from cfromknecht Aug 7, 2019

wpaulino changed the title ~~discovery+rpc: expose graph synced status within GetNetworkInfo~~ discovery+rpc: expose graph synced status within GetInfo Aug 7, 2019

halseth removed their request for review Aug 7, 2019

cfromknecht approved these changes Aug 7, 2019

View changes

Roasbeef approved these changes Aug 13, 2019

View changes

Roasbeef merged commit 4e62e8a into lightningnetwork:master Aug 13, 2019
2 checks passed

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

coverage/coveralls Coverage increased (+0.02%) to 60.782%
Details

wpaulino deleted the wpaulino:is-graph-synced branch Aug 13, 2019

lightningnetwork/lnd

Join GitHub today

discovery+rpc: expose graph synced status within GetInfo #3355

Conversation

wpaulino commented Jul 29, 2019 • edited

This comment has been minimized.

Roasbeef commented Jul 30, 2019

This comment has been minimized.

Roasbeef Jul 30, 2019

This comment has been minimized.

wpaulino Jul 30, 2019

This comment has been minimized.

cfromknecht commented Jul 30, 2019

This comment has been minimized.

wpaulino commented Jul 30, 2019

cfromknecht left a comment

This comment has been minimized.

sergioabril commented Jul 31, 2019

This comment has been minimized.

wpaulino commented Aug 1, 2019

This comment has been minimized.

sergioabril commented Aug 1, 2019

This comment has been minimized.

cfromknecht commented Aug 1, 2019

This comment has been minimized.

wpaulino commented Aug 1, 2019

This comment has been minimized.

cfromknecht commented Aug 2, 2019

This comment has been minimized.

wpaulino commented Aug 2, 2019 • edited

This comment has been minimized.

cfromknecht commented Aug 2, 2019

This comment has been minimized.

wpaulino commented Aug 2, 2019 • edited

This comment has been minimized.

halseth Aug 2, 2019

This comment has been minimized.

wpaulino Aug 6, 2019

This comment has been minimized.

Roasbeef commented Aug 7, 2019

cfromknecht left a comment

Roasbeef left a comment

This comment has been minimized.

githorray commented Aug 13, 2019

This comment has been minimized.

wpaulino commented Aug 14, 2019

wpaulino commented Jul 29, 2019 •

edited

wpaulino commented Aug 2, 2019 •

edited

wpaulino commented Aug 2, 2019 •

edited