Offline RPC #3866

sudo-shashank · 2024-01-11T14:55:58Z

Summary of changes

Changes introduced in this pull request:

Set up offline RPC server: that serves data from CAR file and does not connect to the Filecoin network.
Added serve subcommand: forest-tool api serve [CAR_FILE] command runs an RPC server.
Download a latest snapshot if no snapshot file is provided.
RPC port defaults to 2345 which can be configured using --port flag
CI test

Reference issue to close (if applicable)

Closes #3812

Change checklist

I have performed a self-review of my own code,
I have made corresponding changes to the documentation,
I have added tests that prove my fix is effective or that my feature works (if possible),
I have made sure the CHANGELOG is up-to-date. All user-facing changes should be reflected in this document.

aatifsyed

Could we refactor the creation of the server?

aatifsyed · 2024-01-18T16:54:11Z

scripts/tests/calibnet_rpc_check.sh

+
+# Function to get the number of files in the present working directory
+num_files_here() {
+    find . -type f | wc --lines


nit: this overcounts if I make a file with a newline in the name

fixed, now directly using the path returned from snapshot fetch

aatifsyed · 2024-01-18T16:55:49Z

scripts/tests/calibnet_rpc_check.sh

+    done
+
+    # Check if services on ports 8080 and 8081 have started
+    while ! (nc -z localhost 8080 && nc -z localhost 8081); do


nit: this could be an until loop
also hardcoding the ports here defeats the purpose of using the PORTS array

right, fixed

aatifsyed · 2024-01-18T16:57:37Z

scripts/tests/calibnet_rpc_check.sh

+    result="$($FOREST_TOOL_PATH api compare "$snapshot" --forest /ip4/127.0.0.1/tcp/8080/http --lotus /ip4/127.0.0.1/tcp/8081/http)"
+
+    # Check the result
+    if echo "$result" | tail -n +3 | grep -E -v "\| *(Valid|Timeout) *\| *(Valid|Timeout) *\|"; then


Parsing the output is an odd choice - can we not modify forest-tool api compare to give us useful exit codes?

aatifsyed · 2024-01-18T16:57:58Z

src/rpc/auth_api.rs

@@ -8,7 +8,7 @@ use fvm_ipld_blockstore::Blockstore;
 use jsonrpc_v2::{Data, Error as JsonRpcError, Params};

 /// RPC call to create a new JWT Token
-pub(in crate::rpc) async fn auth_new<DB: Blockstore>(
+pub async fn auth_new<DB: Blockstore>(


aatifsyed · 2024-01-18T16:58:52Z

src/rpc/mod.rs

-mod state_api;
-mod sync_api;
-mod wallet_api;
+pub mod auth_api;


wibni: a module structure that ditches the common suffix use rpc::api::auth rather than rpc::auth_api

i would like to do this in a separate PR

aatifsyed · 2024-01-18T16:59:21Z

src/tool/subcommands/api_cmd.rs


 #[derive(Debug, Subcommand)]
 pub enum ApiCommands {
+    // Serve
+    Serve {


#3812 (comment)

using office rpc name instead of node-less rpc

src/tool/subcommands/api_cmd.rs

aatifsyed · 2024-01-18T17:00:39Z

src/tool/subcommands/api_cmd.rs

+    chain: NetworkChain,
+    rpc_port: u16,
+) -> anyhow::Result<()> {
+    println!("Configuring Node-Less RPC Server");


surprising that we're not using info!(..) logs here

changed to using info logger now

aatifsyed · 2024-01-18T17:02:51Z

src/tool/subcommands/api_cmd.rs

+            .with_method(CHAIN_GET_TIPSET_BY_HEIGHT, chain_get_tipset_by_height::<DB>)
+            .with_method(CHAIN_GET_GENESIS, chain_get_genesis::<DB>)
+            .with_method(CHAIN_GET_TIPSET, chain_get_tipset::<DB>)
+            .with_method(CHAIN_HEAD, chain_head::<DB>)
+            .with_method(CHAIN_GET_BLOCK, chain_get_block::<DB>)
+            .with_method(CHAIN_SET_HEAD, chain_set_head::<DB>)
+            .with_method(CHAIN_GET_MIN_BASE_FEE, chain_get_min_base_fee::<DB>)
+            .with_method(
+                CHAIN_GET_MESSAGES_IN_TIPSET,
+                chain_get_messages_in_tipset::<DB>,
+            )
+            .with_method(CHAIN_GET_PARENT_MESSAGES, chain_get_parent_message::<DB>)
+            .with_method(CHAIN_NOTIFY, chain_notify::<DB>)
+            .with_method(CHAIN_GET_PARENT_RECEIPTS, chain_get_parent_receipts::<DB>)
+            // Message Pool API
+            .with_method(MPOOL_GET_NONCE, mpool_get_nonce::<DB>)
+            .with_method(MPOOL_PENDING, mpool_pending::<DB>)
+            .with_method(MPOOL_PUSH, mpool_push::<DB>)
+            .with_method(MPOOL_PUSH_MESSAGE, mpool_push_message::<DB>)
+            // Sync API
+            .with_method(SYNC_CHECK_BAD, sync_check_bad::<DB>)
+            .with_method(SYNC_MARK_BAD, sync_mark_bad::<DB>)
+            .with_method(SYNC_STATE, sync_state::<DB>)
+            // Wallet API
+            .with_method(WALLET_BALANCE, wallet_balance::<DB>)
+            .with_method(WALLET_DEFAULT_ADDRESS, wallet_default_address::<DB>)
+            .with_method(WALLET_EXPORT, wallet_export::<DB>)
+            .with_method(WALLET_HAS, wallet_has::<DB>)
+            .with_method(WALLET_IMPORT, wallet_import::<DB>)
+            .with_method(WALLET_LIST, wallet_list::<DB>)
+            .with_method(WALLET_NEW, wallet_new::<DB>)
+            .with_method(WALLET_SET_DEFAULT, wallet_set_default::<DB>)
+            .with_method(WALLET_SIGN, wallet_sign::<DB>)
+            .with_method(WALLET_VERIFY, wallet_verify)
+            .with_method(WALLET_DELETE, wallet_delete::<DB>)
+            // State API
+            .with_method(STATE_CALL, state_call::<DB>)
+            .with_method(STATE_REPLAY, state_replay::<DB>)
+            .with_method(STATE_NETWORK_NAME, state_network_name::<DB>)
+            .with_method(STATE_NETWORK_VERSION, state_get_network_version::<DB>)
+            .with_method(STATE_ACCOUNT_KEY, state_account_key::<DB>)
+            .with_method(STATE_LOOKUP_ID, state_lookup_id::<DB>)
+            .with_method(STATE_GET_ACTOR, state_get_actor::<DB>)
+            .with_method(STATE_MARKET_BALANCE, state_market_balance::<DB>)
+            .with_method(STATE_MARKET_DEALS, state_market_deals::<DB>)
+            .with_method(STATE_MINER_INFO, state_miner_info::<DB>)
+            .with_method(MINER_GET_BASE_INFO, miner_get_base_info::<DB>)
+            .with_method(STATE_MINER_ACTIVE_SECTORS, state_miner_active_sectors::<DB>)
+            .with_method(STATE_MINER_SECTOR_COUNT, state_miner_sector_count::<DB>)
+            .with_method(STATE_MINER_FAULTS, state_miner_faults::<DB>)
+            .with_method(STATE_MINER_RECOVERIES, state_miner_recoveries::<DB>)
+            .with_method(STATE_MINER_POWER, state_miner_power::<DB>)
+            .with_method(STATE_MINER_DEADLINES, state_miner_deadlines::<DB>)
+            .with_method(STATE_LIST_MINERS, state_list_miners::<DB>)
+            .with_method(
+                STATE_MINER_PROVING_DEADLINE,
+                state_miner_proving_deadline::<DB>,
+            )
+            .with_method(STATE_GET_RECEIPT, state_get_receipt::<DB>)
+            .with_method(STATE_WAIT_MSG, state_wait_msg::<DB>)
+            .with_method(STATE_SEARCH_MSG, state_search_msg::<DB>)
+            .with_method(STATE_SEARCH_MSG_LIMITED, state_search_msg_limited::<DB>)
+            .with_method(STATE_FETCH_ROOT, state_fetch_root::<DB>)
+            .with_method(
+                STATE_GET_RANDOMNESS_FROM_TICKETS,
+                state_get_randomness_from_tickets::<DB>,
+            )
+            .with_method(
+                STATE_GET_RANDOMNESS_FROM_BEACON,
+                state_get_randomness_from_beacon::<DB>,
+            )
+            .with_method(STATE_READ_STATE, state_read_state::<DB>)
+            .with_method(STATE_CIRCULATING_SUPPLY, state_circulating_supply::<DB>)
+            .with_method(STATE_SECTOR_GET_INFO, state_sector_get_info::<DB>)
+            .with_method(
+                STATE_VM_CIRCULATING_SUPPLY_INTERNAL,
+                state_vm_circulating_supply_internal::<DB>,
+            )
+            .with_method(MSIG_GET_AVAILABLE_BALANCE, msig_get_available_balance::<DB>)
+            .with_method(MSIG_GET_PENDING, msig_get_pending::<DB>)
+            // Gas API
+            .with_method(GAS_ESTIMATE_FEE_CAP, gas_estimate_fee_cap::<DB>)
+            .with_method(GAS_ESTIMATE_GAS_LIMIT, gas_estimate_gas_limit::<DB>)
+            .with_method(GAS_ESTIMATE_GAS_PREMIUM, gas_estimate_gas_premium::<DB>)
+            .with_method(GAS_ESTIMATE_MESSAGE_GAS, gas_estimate_message_gas::<DB>)
+            // Common API
+            .with_method(VERSION, move || {
+                version(block_delay, FOREST_VERSION_STRING.as_str())
+            })
+            .with_method(SESSION, session)
+            .with_method(START_TIME, start_time::<DB>)
+            // Node API
+            .with_method(NODE_STATUS, node_status::<DB>)
+            // Eth API
+            .with_method(ETH_ACCOUNTS, eth_accounts)
+            .with_method(ETH_BLOCK_NUMBER, eth_block_number::<DB>)
+            .with_method(ETH_CHAIN_ID, eth_chain_id::<DB>)
+            .with_method(ETH_GAS_PRICE, eth_gas_price::<DB>)
+            .with_method(ETH_GET_BALANCE, eth_get_balance::<DB>)
+            .finish_unwrapped(),


have I seen this code before? Could we refactor?

Yeah, it would be really great to have a single "helper" method for all the occurrences of RPC definitions, assuming they are all the same. If not - perhaps whatever is shared is a good start.

Something like:

fn rpc_methods(server: Arc<Server>) or otherwise we could define a custom trait and implement it for the Server struct, see extension traits for inspiration.

Ah, I can see now that this is RPC v1. As opposed to v0 or ws. Curious if there is already a definition for that elsewhere.

using start_rpc method now from rpc crate

.github/workflows/forest.yml

hanabi1224 · 2024-01-19T08:42:51Z

src/tool/subcommands/api_cmd.rs

+        );
+        let snapshot_url =
+            crate::cli_shared::snapshot::stable_url(TrustedVendor::default(), &chain)?;
+        let tmp_snapshot_path = tempfile::Builder::new().tempdir()?.into_path();


/tmp typically has 10GB size limit which is insufficient for downloading a mainnet snapshot

I think we could just keep the logic simple, ask users to download a snapshot via forest-tool snapshot fetch

Or use logic similar to that of the running node? Seems laborious having to download snapshots manually..

I agree with roman, I think we should keep the automatic snapshot download feature if a snapshot is not provided

Yeah, but you have to take into account Hailong's comment about tmp dir size.

using project dir now

CHANGELOG.md

hanabi1224 · 2024-01-29T12:28:45Z

src/tool/subcommands/api_cmd.rs

+    let (snapshot_file, snapshot_path) = if let Some(path) = snapshot_path_opt {
+        (File::open(&path).await?, path)
+    } else {
+        warn!(


Shall we add some prompt before downloading? Like in other places

yes i'll add a similar prompt

Thanks, then I guess we also need a similar --force or --yes option to bypass the prompt

Impl node less rpc

892e9a5

sudo-shashank changed the title ~~Node less RPC~~ Node-less RPC Jan 11, 2024

sudo-shashank and others added 15 commits January 16, 2024 16:39

fix

a257bef

Set SyncStage to idle and Cleanup

8b745c9

fmt

8e1beaa

Merge branch 'main' into shashank/node-less-rpc

447303d

fix and add new methods

ed75192

add new rpc

731e154

lint-fix

03de8c8

Merge branch 'main' into shashank/node-less-rpc

200c84d

add ci test

60256eb

fix check rpc script

f49c717

updated changelog

8764c90

lint fix

a47ec6f

fix

fd47b3b

Merge branch 'main' into shashank/node-less-rpc

d0c68fa

Merge branch 'main' into shashank/node-less-rpc

fcb1ad3

sudo-shashank marked this pull request as ready for review January 18, 2024 13:47

sudo-shashank requested a review from a team as a code owner January 18, 2024 13:47

sudo-shashank requested review from ruseinov and aatifsyed and removed request for a team January 18, 2024 13:47

aatifsyed requested changes Jan 18, 2024

View reviewed changes

hanabi1224 reviewed Jan 19, 2024

View reviewed changes

.github/workflows/forest.yml Outdated Show resolved Hide resolved

hanabi1224 reviewed Jan 19, 2024

View reviewed changes

sudo-shashank changed the title ~~Node-less RPC~~ Offline RPC Jan 23, 2024

sudo-shashank added 4 commits January 23, 2024 17:38

refactored api serve subcommand

8de82d7

use logger

4d8b7f4

fix logger

2a6adb0

Merge branch 'main' into shashank/node-less-rpc

6e0e8a0

sudo-shashank and others added 7 commits January 23, 2024 20:21

Updated offline rpc test script

9d17dfd

lint fix

a88cb56

fix

0540c17

fix typo

854325e

limit tipset in compare

2cad4ba

cleanup

2e4c090

Merge branch 'main' into shashank/node-less-rpc

99627e0

sudo-shashank requested review from hanabi1224 and aatifsyed January 24, 2024 12:56

sudo-shashank and others added 4 commits January 29, 2024 10:34

Merge branch 'main' into shashank/node-less-rpc

366eff8

Merge branch 'main' into shashank/node-less-rpc

0b05045

Merge branch 'main' into shashank/node-less-rpc

0f82ed6

treat timeout as success

0ea032e

hanabi1224 reviewed Jan 29, 2024

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

hanabi1224 reviewed Jan 29, 2024

View reviewed changes

sudo-shashank added 4 commits January 29, 2024 17:59

fix changelog

4595a4e

fmt

59a55d2

fmt

7f21806

add auto snapshot download option

0bf99d0

sudo-shashank requested a review from hanabi1224 January 30, 2024 09:00

sudo-shashank and others added 4 commits January 30, 2024 14:36

Merge branch 'main' into shashank/node-less-rpc

1b4e0c8

fix peek method call

79002c4

Merge branch 'main' into shashank/node-less-rpc

2e03771

Merge branch 'main' into shashank/node-less-rpc

e16498e

hanabi1224 approved these changes Jan 30, 2024

View reviewed changes

aatifsyed approved these changes Jan 30, 2024

View reviewed changes

aatifsyed added this pull request to the merge queue Jan 30, 2024

Merged via the queue into main with commit 7d22ef0 Jan 30, 2024
28 checks passed

aatifsyed deleted the shashank/node-less-rpc branch January 30, 2024 14:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Offline RPC #3866

Offline RPC #3866

sudo-shashank commented Jan 11, 2024 •

edited

aatifsyed left a comment

aatifsyed Jan 18, 2024

sudo-shashank Jan 23, 2024

aatifsyed Jan 18, 2024

sudo-shashank Jan 23, 2024

aatifsyed Jan 18, 2024

sudo-shashank Jan 23, 2024

aatifsyed Jan 18, 2024

aatifsyed Jan 18, 2024

sudo-shashank Jan 23, 2024

aatifsyed Jan 18, 2024

sudo-shashank Jan 23, 2024

aatifsyed Jan 18, 2024

sudo-shashank Jan 23, 2024 •

edited

aatifsyed Jan 18, 2024

ruseinov Jan 22, 2024

ruseinov Jan 22, 2024

sudo-shashank Jan 23, 2024

hanabi1224 Jan 19, 2024

hanabi1224 Jan 19, 2024

ruseinov Jan 22, 2024 •

edited

sudo-shashank Jan 23, 2024

ruseinov Jan 23, 2024

sudo-shashank Jan 29, 2024

hanabi1224 Jan 29, 2024 •

edited

sudo-shashank Jan 29, 2024

hanabi1224 Jan 30, 2024

sudo-shashank Jan 30, 2024

Offline RPC #3866

Offline RPC #3866

Conversation

sudo-shashank commented Jan 11, 2024 • edited

Summary of changes

Reference issue to close (if applicable)

Other information and links

Change checklist

aatifsyed left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sudo-shashank Jan 23, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruseinov Jan 22, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hanabi1224 Jan 29, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sudo-shashank commented Jan 11, 2024 •

edited

sudo-shashank Jan 23, 2024 •

edited

ruseinov Jan 22, 2024 •

edited

hanabi1224 Jan 29, 2024 •

edited