Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: On Demand Peer Discovery based on shard and service #834

Merged
merged 28 commits into from
Nov 7, 2023

Conversation

chaitanyaprem
Copy link
Collaborator

@chaitanyaprem chaitanyaprem commented Oct 24, 2023

Description

Ability to discover peers on demand based on shard and service. This would help lightclients and relay nodes during new shard subscription in reducing wait time in discovery.

Changes

  • move discovery and common service to separate package to remove package inter-dependencies
  • Refactor code to remove import cycle dependency
  • PeerManager to use discoveryv5 for on-demand peer-discovery
  • Use on-demand peer discovery in relay subscription to new topics
  • Service Peer Selection to trigger on demand discv5 to find peers specific to shard and service
  • Specify protocols supported while adding discovered peers based on its ENR
  • Tests to validate peer discovery and discovery during peer selection

Tests

@status-im-auto
Copy link

status-im-auto commented Oct 24, 2023

Jenkins Builds

Click to see older builds (128)
Commit #️⃣ Finished (UTC) Duration Platform Result
✔️ b07334e #1 2023-10-24 22:34:55 ~1 min nix-flake 📄log
✔️ b07334e #1 2023-10-24 22:35:24 ~2 min linux 📦deb
✔️ b07334e #1 2023-10-24 22:36:19 ~3 min tests 📄log
✔️ b07334e #1 2023-10-24 22:36:25 ~3 min tests 📄log
✔️ b07334e #1 2023-10-24 22:36:43 ~3 min android 📦tgz
✔️ b07334e #1 2023-10-24 22:37:25 ~4 min ios 📦tgz
✔️ 0eb933b #2 2023-10-26 18:53:22 ~1 min linux 📦deb
✖️ 0eb933b #2 2023-10-26 18:53:23 ~1 min tests 📄log
✔️ 0eb933b #2 2023-10-26 18:54:05 ~1 min nix-flake 📄log
✔️ 0eb933b #2 2023-10-26 18:55:38 ~3 min tests 📄log
✔️ 0eb933b #2 2023-10-26 18:56:03 ~3 min android 📦tgz
✔️ 0eb933b #2 2023-10-26 18:56:21 ~4 min ios 📦tgz
✔️ 0eb933b #3 2023-10-26 18:57:42 ~3 min tests 📄log
✔️ b6f5a8a #3 2023-10-26 18:58:01 ~1 min nix-flake 📄log
✔️ b6f5a8a #3 2023-10-26 18:58:10 ~2 min tests 📄log
✔️ b6f5a8a #3 2023-10-26 18:58:30 ~2 min linux 📦deb
✔️ b6f5a8a #4 2023-10-26 19:00:02 ~2 min tests 📄log
✔️ b6f5a8a #3 2023-10-26 19:00:11 ~4 min android 📦tgz
✔️ b6f5a8a #3 2023-10-26 19:00:31 ~4 min ios 📦tgz
✖️ d994aca #4 2023-10-27 02:14:54 ~23 sec tests 📄log
✖️ d994aca #5 2023-10-27 02:14:56 ~23 sec tests 📄log
✔️ d994aca #4 2023-10-27 02:16:24 ~1 min nix-flake 📄log
✔️ d994aca #4 2023-10-27 02:16:48 ~2 min linux 📦deb
✔️ d994aca #4 2023-10-27 02:17:59 ~3 min android 📦tgz
✔️ d994aca #4 2023-10-27 02:20:22 ~5 min ios 📦tgz
✖️ f4fd8aa #6 2023-10-27 18:20:25 ~26 sec tests 📄log
✖️ f4fd8aa #5 2023-10-27 18:20:27 ~28 sec tests 📄log
✔️ f4fd8aa #5 2023-10-27 18:21:07 ~1 min linux 📦deb
✔️ f4fd8aa #5 2023-10-27 18:21:55 ~1 min nix-flake 📄log
✔️ f4fd8aa #5 2023-10-27 18:24:27 ~4 min android 📦tgz
✔️ f4fd8aa #5 2023-10-27 18:24:47 ~4 min ios 📦tgz
✖️ 27a8d46 #6 2023-10-27 19:20:15 ~23 sec tests 📄log
✖️ 27a8d46 #7 2023-10-27 19:20:19 ~24 sec tests 📄log
✔️ 27a8d46 #6 2023-10-27 19:20:54 ~1 min linux 📦deb
✔️ 27a8d46 #6 2023-10-27 19:21:48 ~1 min nix-flake 📄log
✔️ 27a8d46 #6 2023-10-27 19:23:11 ~3 min android 📦tgz
✔️ 27a8d46 #6 2023-10-27 19:23:35 ~3 min ios 📦tgz
✔️ 36bcea6 #7 2023-10-27 20:55:02 ~1 min linux 📦deb
✔️ 36bcea6 #7 2023-10-27 20:55:46 ~1 min tests 📄log
✔️ 36bcea6 #7 2023-10-27 20:55:48 ~1 min nix-flake 📄log
✔️ 36bcea6 #8 2023-10-27 20:56:16 ~2 min tests 📄log
✔️ 36bcea6 #7 2023-10-27 20:57:19 ~3 min android 📦tgz
✔️ 36bcea6 #7 2023-10-27 20:58:10 ~4 min ios 📦tgz
✔️ 14d6983 #8 2023-10-27 21:18:45 ~1 min linux 📦deb
✔️ 14d6983 #8 2023-10-27 21:19:34 ~1 min nix-flake 📄log
✔️ 14d6983 #8 2023-10-27 21:19:35 ~1 min tests 📄log
✔️ 14d6983 #9 2023-10-27 21:20:01 ~2 min tests 📄log
✔️ 14d6983 #8 2023-10-27 21:21:11 ~3 min android 📦tgz
✔️ 14d6983 #8 2023-10-27 21:21:46 ~4 min ios 📦tgz
bea2a1f #9 2023-11-02 05:43:24 ~21 sec linux 📄log
✖️ bea2a1f #10 2023-11-02 05:43:27 ~20 sec tests 📄log
✖️ bea2a1f #9 2023-11-02 05:43:28 ~22 sec tests 📄log
✖️ bea2a1f #9 2023-11-02 05:43:56 ~52 sec nix-flake 📄log
bea2a1f #9 2023-11-02 05:45:55 ~2 min android 📄log
bea2a1f #9 2023-11-02 05:46:25 ~3 min ios 📄log
✖️ bb95a6f #10 2023-11-03 06:44:00 ~1 min tests 📄log
✔️ bb95a6f #10 2023-11-03 06:44:19 ~1 min linux 📦deb
✔️ bb95a6f #10 2023-11-03 06:44:21 ~1 min nix-flake 📄log
✔️ bb95a6f #11 2023-11-03 06:44:53 ~2 min tests 📄log
✔️ bb95a6f #10 2023-11-03 06:46:36 ~4 min android 📦tgz
✔️ bb95a6f #10 2023-11-03 06:46:41 ~4 min ios 📦tgz
✔️ bb95a6f #11 2023-11-03 07:18:02 ~3 min tests 📄log
✔️ 8a94cc8 #11 2023-11-03 07:32:57 ~1 min linux 📦deb
✔️ 8a94cc8 #11 2023-11-03 07:33:43 ~1 min nix-flake 📄log
✔️ 8a94cc8 #12 2023-11-03 07:34:00 ~2 min tests 📄log
✔️ 8a94cc8 #12 2023-11-03 07:34:12 ~2 min tests 📄log
✔️ 8a94cc8 #11 2023-11-03 07:35:09 ~3 min ios 📦tgz
✔️ 8a94cc8 #11 2023-11-03 07:35:28 ~3 min android 📦tgz
✔️ c5d814b #12 2023-11-03 10:43:48 ~1 min linux 📦deb
✔️ c5d814b #12 2023-11-03 10:44:19 ~1 min nix-flake 📄log
✔️ c5d814b #13 2023-11-03 10:44:30 ~2 min tests 📄log
✔️ c5d814b #13 2023-11-03 10:45:52 ~3 min tests 📄log
✔️ c5d814b #12 2023-11-03 10:45:57 ~3 min android 📦tgz
✔️ c5d814b #12 2023-11-03 10:46:04 ~3 min ios 📦tgz
✔️ 7549987 #13 2023-11-03 15:09:06 ~2 min nix-flake 📄log
✔️ 7549987 #13 2023-11-03 15:09:58 ~3 min ios 📦tgz
✔️ 7549987 #13 2023-11-03 15:10:29 ~3 min linux 📦deb
✔️ 7549987 #13 2023-11-03 15:10:36 ~4 min android 📦tgz
✔️ 7549987 #14 2023-11-03 15:12:34 ~5 min tests 📄log
✔️ 7549987 #14 2023-11-03 15:13:05 ~6 min tests 📄log
✔️ 9093816 #14 2023-11-06 04:37:54 ~1 min linux 📦deb
✔️ 9093816 #14 2023-11-06 04:37:59 ~1 min nix-flake 📄log
✔️ 9093816 #15 2023-11-06 04:38:21 ~2 min tests 📄log
✔️ 9093816 #15 2023-11-06 04:38:33 ~2 min tests 📄log
✔️ 9093816 #14 2023-11-06 04:40:21 ~4 min ios 📦tgz
✔️ 9093816 #14 2023-11-06 04:40:28 ~4 min android 📦tgz
✔️ 3027659 #15 2023-11-06 04:40:39 ~1 min linux 📦deb
✔️ 3027659 #15 2023-11-06 04:40:46 ~1 min nix-flake 📄log
✔️ 3027659 #16 2023-11-06 04:41:01 ~2 min tests 📄log
✔️ 3027659 #16 2023-11-06 04:41:15 ~2 min tests 📄log
✔️ 3027659 #15 2023-11-06 04:42:43 ~3 min ios 📦tgz
✔️ 3027659 #15 2023-11-06 04:43:55 ~3 min android 📦tgz
✔️ 25a74ab #16 2023-11-06 06:06:31 ~1 min linux 📦deb
✖️ 25a74ab #17 2023-11-06 06:06:37 ~1 min tests 📄log
✖️ 25a74ab #17 2023-11-06 06:06:51 ~1 min tests 📄log
✔️ 25a74ab #16 2023-11-06 06:07:10 ~1 min nix-flake 📄log
✔️ 25a74ab #16 2023-11-06 06:08:54 ~3 min android 📦tgz
✔️ 25a74ab #16 2023-11-06 06:09:11 ~3 min ios 📦tgz
✔️ 6a620ff #17 2023-11-06 06:46:17 ~1 min linux 📦deb
✔️ 6a620ff #17 2023-11-06 06:46:54 ~1 min nix-flake 📄log
✔️ 6a620ff #18 2023-11-06 06:47:09 ~2 min tests 📄log
✔️ 6a620ff #18 2023-11-06 06:47:26 ~2 min tests 📄log
✔️ 6a620ff #17 2023-11-06 06:48:22 ~3 min ios 📦tgz
✔️ 6a620ff #17 2023-11-06 06:48:25 ~3 min android 📦tgz
✔️ 2197a1a #18 2023-11-06 06:53:39 ~1 min linux 📦deb
✔️ 2197a1a #19 2023-11-06 06:53:44 ~1 min tests 📄log
✔️ 2197a1a #18 2023-11-06 06:54:28 ~1 min nix-flake 📄log
✔️ 2197a1a #19 2023-11-06 06:54:54 ~2 min tests 📄log
✔️ 2197a1a #18 2023-11-06 06:55:40 ~3 min android 📦tgz
✔️ 2197a1a #18 2023-11-06 06:56:03 ~3 min ios 📦tgz
✔️ a947759 #19 2023-11-06 07:22:51 ~1 min linux 📦deb
✔️ a947759 #19 2023-11-06 07:23:38 ~1 min nix-flake 📄log
✔️ a947759 #20 2023-11-06 07:23:52 ~2 min tests 📄log
✔️ a947759 #20 2023-11-06 07:24:09 ~2 min tests 📄log
✔️ a947759 #19 2023-11-06 07:25:06 ~3 min ios 📦tgz
✔️ a947759 #19 2023-11-06 07:25:14 ~3 min android 📦tgz
✔️ 7e349d7 #20 2023-11-06 10:32:39 ~1 min linux 📦deb
✔️ 7e349d7 #20 2023-11-06 10:33:18 ~1 min nix-flake 📄log
✔️ 7e349d7 #21 2023-11-06 10:33:38 ~2 min tests 📄log
✔️ 7e349d7 #21 2023-11-06 10:33:55 ~2 min tests 📄log
✔️ 7e349d7 #20 2023-11-06 10:34:53 ~3 min ios 📦tgz
✔️ 7e349d7 #20 2023-11-06 10:35:00 ~3 min android 📦tgz
✔️ afdbd48 #21 2023-11-07 15:01:27 ~3 min ios 📦tgz
✔️ afdbd48 #21 2023-11-07 15:03:47 ~6 min linux 📦deb
✖️ afdbd48 #22 2023-11-07 15:03:57 ~6 min tests 📄log
✖️ afdbd48 #22 2023-11-07 15:04:05 ~6 min tests 📄log
✔️ afdbd48 #21 2023-11-07 15:04:12 ~6 min nix-flake 📄log
✔️ afdbd48 #21 2023-11-07 15:04:56 ~7 min android 📦tgz
Commit #️⃣ Finished (UTC) Duration Platform Result
46bda19 #22 2023-11-07 16:23:33 ~18 sec linux 📄log
✖️ 46bda19 #23 2023-11-07 16:23:37 ~21 sec tests 📄log
✖️ 46bda19 #23 2023-11-07 16:23:40 ~24 sec tests 📄log
✖️ 46bda19 #22 2023-11-07 16:24:16 ~1 min nix-flake 📄log
46bda19 #22 2023-11-07 16:25:48 ~2 min android 📄log
46bda19 #22 2023-11-07 16:26:09 ~2 min ios 📄log
✔️ e07e5d1 #23 2023-11-07 17:08:54 ~1 min linux 📦deb
✔️ e07e5d1 #23 2023-11-07 17:09:39 ~1 min nix-flake 📄log
✔️ e07e5d1 #24 2023-11-07 17:10:01 ~2 min tests 📄log
✔️ e07e5d1 #24 2023-11-07 17:10:20 ~2 min tests 📄log
✔️ e07e5d1 #23 2023-11-07 17:11:12 ~3 min ios 📦tgz
✔️ e07e5d1 #23 2023-11-07 17:11:25 ~3 min android 📦tgz

waku/v2/peermanager/peer_discovery.go Dismissed Show dismissed Hide dismissed
@chaitanyaprem chaitanyaprem marked this pull request as ready for review November 3, 2023 07:38
@chaitanyaprem chaitanyaprem changed the title Feat: On Demand Peer Discovery based on shard and service feat: On Demand Peer Discovery based on shard and service Nov 3, 2023
return peerID, nil
} else if !errors.Is(err, ErrNoPeersAvailable) {
pm.logger.Debug("could not retrieve random peer from slot", zap.String("protocol", string(criteria.Proto)),
zap.String("pubsubTopic", criteria.PubsubTopic), zap.Error(err))

Check failure

Code scanning / CodeQL

Log entries created from user input High

This log entry depends on a
user-provided value
.
if err == nil {
return peerID, nil
} else {
pm.logger.Debug("Discovering peers by pubsubTopic", zap.String("pubsubTopic", pubsubTopic))

Check failure

Code scanning / CodeQL

Log entries created from user input High

This log entry depends on a
user-provided value
.
waku/v2/protocol/filter/server.go Outdated Show resolved Hide resolved
waku/v2/peermanager/peer_discovery.go Outdated Show resolved Hide resolved
waku/v2/peermanager/peer_discovery.go Show resolved Hide resolved
waku/v2/peermanager/peer_manager.go Outdated Show resolved Hide resolved
pm.logger.Error("failed to discover and conenct to peers", zap.Error(err))
}
} else {
pm.logger.Debug("failed to convert pubsub topic to shard as topic is named pubsubTopic", zap.String("topic", pubsubTopic))

Check failure

Code scanning / CodeQL

Log entries created from user input High

This log entry depends on a
user-provided value
.
@chaitanyaprem chaitanyaprem merged commit 3226def into master Nov 7, 2023
9 of 11 checks passed
@chaitanyaprem chaitanyaprem deleted the feat/on-demand-disc branch November 7, 2023 17:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

None yet

3 participants