Network debug page assumes that debug RPC port is 3030

[saketh-are]

The NetworkInfo debug page assumes that the debug RPC port for connected peers is always 3030, and that the RPC address is the same as the node's network address:

nearcore/chain/jsonrpc/res/network_info.js

Line 43 in 2a2b5d6

href: "http://" + peer_addr.replace(/:.*/, ":3030/debug"),

Localnet nodes spawned using nearup work differently (link):

data['rpc']['addr'] = f'0.0.0.0:{3030 + num_nodes}'
data['network']['addr'] = f'0.0.0.0:{24567 + num_nodes}'

The NetworkInfo page should be modified to handle localnet setups correctly.

This new debug UI will need to be updated as well:

nearcore/tools/debug-ui/src/utils.tsx

Line 48 in 907538f

href={"http://localhost:3000/" + peer_addr.replace(/:.*/, "/")}>

[ghost]

The main issue is that the href (link that appears once you click on it), will always go to 3030/debug instead of the proper values of 3031/debug, 3032/debug etc.

[ghost]

20230404: Debug Page RPC Port

Overview

Github Issue

• Network debug page assumes that debug RPC port is 3030 #8583

Code Commit

• Fix debug rpc ports localnet #8583 #8866

Google Doc

• 20230404 Debug Page RPC Port

Problem Statement

Debug pages work by a separate web server making a RPC call to the node it is on. However, it is currently hard-coding the RPC address to 3030, which does not work for localnet. As a result, developers have to manually type the RPC address to view the debug pages from each node’s perspective of debugging information in localnet.

Background: Network Address and RPC Address

There are currently 2 types of addresses

• Network address
  • These are the socket address used by each node in the network to communicate with each other for Near Protocol to run
• RPC address
  • These are the socket address used by clients of Near Protocol to use Near’s API

Code assumes each node always uses port numbers:

• 3030 as the rpc address
• 24567 as the network address

However, this assumption is only practical if each node runs on a separate computer as they do not share port number space.

For localnet, we run multiple processes on the same computer, hence, the ports cannot all be 3030 or 24567 as it will result in a clash with other processes. Each node on localnet set up their addresses on different ports with a simple increment:

• RPC address
  • 3030 + nodeNumber, where nodeNumber = {0, 1, 2, ..., number of Nodes - 1)
• Network address
  • 24567 + nodeNumber, where nodeNumber = {0, 1, 2, ..., number of Nodes - 1)

Technical Challenges

Lack of peer’s RPC address

Detect other node’s RPC: The main issue here is there’s currently no information being sent between nodes on their rpc address as they only share their network addresses

Each node only contains minimal information from other nodes (to reduce network bandwidth)

• peer identifier (uniqueness guaranteed based on monotonically incrementing counter)
• network address
• near account identifier (uniqueness guaranteed during account creation)

Possible Solutions

Feel free to skip and jump straight to the chosen solution: Solution 5

Solution 1: Always Include rpc port in Peer Info

Currently, the PeerInfo does not include rpc port, simply include it in order for other nodes to be aware

• nearcore/chain/network/src/network_protocol/peer.rs

  Lines 13 to 19 in a5ca954

  	/// Peer information.
  	#[derive(borsh::BorshSerialize, borsh::BorshDeserialize, Clone, Debug, Eq, PartialEq, Hash)]
  	pub struct PeerInfo {
  	pub id: PeerId,
  	pub addr: Option<SocketAddr>,
  	pub account_id: Option<AccountId>,
  	}

Drawbacks:

• This will introduce both code and network bandwidth

Solution 2: Optionally Include rpc port in Peer Info

Similar to Solution1, but only include rpc port into PeerInfo as an optional parameter if it’s localnet.

Drawbacks:

• This will introduce a lot of optional codes that should only run for localnet, and conditional to detect if it’s local net, which isn’t useful for production environments and will complicate the code base.

Solution 3: Expose API for RPC Port

Create a new API to be able to fetch each node’s RPC port in the status code

• nearcore/core/primitives/src/views.rs

  Lines 638 to 679 in 700ec29

  	#[derive(serde::Serialize, serde::Deserialize, Debug)]
  	pub struct DetailedDebugStatus {
  	pub network_info: NetworkInfoView,
  	pub sync_status: String,
  	pub catchup_status: Vec<CatchupStatusView>,
  	pub current_head_status: BlockStatusView,
  	pub current_header_head_status: BlockStatusView,
  	pub block_production_delay_millis: u64,
  	}
  	// TODO: add more information to status.
  	#[derive(serde::Serialize, serde::Deserialize, Debug)]
  	pub struct StatusResponse {
  	/// Binary version.
  	pub version: Version,
  	/// Unique chain id.
  	pub chain_id: String,
  	/// Currently active protocol version.
  	pub protocol_version: u32,
  	/// Latest protocol version that this client supports.
  	pub latest_protocol_version: u32,
  	/// Address for RPC server. None if node doesn’t have RPC endpoint enabled.
  	#[serde(skip_serializing_if = "Option::is_none")]
  	pub rpc_addr: Option<String>,
  	/// Current epoch validators.
  	pub validators: Vec<ValidatorInfo>,
  	/// Sync status of the node.
  	pub sync_info: StatusSyncInfo,
  	/// Validator id of the node
  	pub validator_account_id: Option<AccountId>,
  	/// Public key of the validator.
  	pub validator_public_key: Option<PublicKey>,
  	/// Public key of the node.
  	pub node_public_key: PublicKey,
  	/// Deprecated; same as `validator_public_key` which you should use instead.
  	pub node_key: Option<PublicKey>,
  	/// Uptime of the node.
  	pub uptime_sec: i64,
  	/// Information about last blocks, network, epoch and chain & chunk info.
  	#[serde(skip_serializing_if = "Option::is_none")]
  	pub detailed_debug_status: Option<DetailedDebugStatus>,
  	}

Also, a new technical challenge would be the chicken-egg problem. The node will need access to another node’s json RPC port in order to call it’s json RPC port.

Drawbacks:

• This is a lot of effort and not worth it for this task, which is considered low priority.

Solution 4: Hard-code paths to localnet’s configurations

During localnet, all nodes config.json will be available for read on the same computer.

The config.json contains the rpc ports of each node

• nearcore/nearcore/src/config.rs

  Lines 621 to 623 in 7d00fef

  	pub network_config: NetworkConfig,
  	#[cfg(feature = "json_rpc")]
  	pub rpc_config: Option<RpcConfig>,

Drawbacks:

• This introduces a new technical challenge of figuring out where each node’s config.json are.

Solution 5: Assume default ports are used

Just assume the ports for localnet are always set as:

• RPC address
  • 3030 + nodeNumber, where nodeNumber = {0, 1, 2, ..., number of Nodes - 1)
• Network address
  • 24567 + nodeNumber, where nodeNumber = {0, 1, 2, ..., number of Nodes - 1)

Then, perform simple arithmetic to infer a node’s RPC address from it’s network address

// Simple Arithmetic Code
default_network_port = 24567
default_rpc_port = 3030
inferred_peer_rpc_port = default_rpc_port + (peer_network_port - default_network_port)

// Full code
// To mitigate the case where mainnet node could use a different rpc port while maintaining the default port, we set a conditional to only perform the above arithmetic for localnet 
node_num = 0 // first node is 0
if self.ip_address == peer.ip_address
{
  // To mitigate the case where mainnet node could use a different rpc port while maintaining the default port, set a conditional to only perform the above arithmetic for localnet 
  default_network_port = 24567
  default_rpc_port = 3030
  node_num = peer_network_port - default_network_port
}
inferred_peer_rpc_port = default_rpc_port + node_num

Drawbacks:

• This requires assuming the rpc address and network address are set as such and would break otherwise

Mitigations for drawback

• This code is mainly for localnet testing on debug pages and does not affect the actual core near protocol. Hence, even if it breaks, a fix would simply be on the front end and does not require a re-deployment of any kind (new debug pages on localnet are able to debug nodes on mainnet
• Check that it’s localnet by ensuring the ip address (without the port) matches, which is unique to localnet

Tests

I only performed manual testing as there’s no quick way to write unit tests for our current frontend debug pages code. Also, the old debug pages are going to be deprecate so no point investing time into testing the old debug page.

// Ran linter on new debug page (linter not set-up for old debug page)
npm run lint

// Open both old and new debug pages and manually click on the links on network_info page to verify they work
// Start up localnet
cd nearcore
make debug
nearup run localnet --binary-path ~/Github/near/nearcore/target/debug/
// Open old debug page
open http://localhost:3030/debug
// Open new debug page
cd nearcore/tools/debug-ui
npm install
npm start
open http://localhost:3000/localhost:3030/cluster

[ghost]

Closing as code is merged