Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed to fetch the latest block #837

Closed
boundless-forest opened this issue Apr 26, 2024 · 8 comments
Closed

Failed to fetch the latest block #837

boundless-forest opened this issue Apr 26, 2024 · 8 comments

Comments

@boundless-forest
Copy link

boundless-forest commented Apr 26, 2024

I am using ponder to index the evm logs based on our substrate ethereum compatible chain. However, I have ecountered an issue that fails to fetch the latest block.

  • Ponder version: v0.4.10

The ponder logs:

9:44:14 AM INFO  sync       Completed historical sync across all networks
9:44:14 AM INFO  indexing   Completed historical indexing
9:44:14 AM INFO  server     Started responding as healthy
9:47:01 AM ERROR Request failed, request: {"method":"eth_getBlockByNumber","params":["latest",true]}
TimeoutError: The request took too long to respond.

URL: http://127.0.0.1:9944
Request body: {"method":"eth_getBlockByNumber","params":["latest",true]}

Details: The request timed out.
Version: viem@1.21.3
    at Object.http (file:///home/bear/coding/rust-space/ormponder/node_modules/viem/utils/rpc.ts:115:24)
    at fn (file:///home/bear/coding/rust-space/ormponder/node_modules/viem/clients/transports/http.ts:107:28)
    at request (file:///home/bear/coding/rust-space/ormponder/node_modules/viem/clients/transports/http.ts:109:45)
    at withRetry.delay.count.count (file:///home/bear/coding/rust-space/ormponder/node_modules/viem/utils/buildRequest.ts:129:24)
    at attemptRetry (file:///home/bear/coding/rust-space/ormponder/node_modules/viem/utils/promise/withRetry.ts:37:28)
    at retry (file:///home/bear/coding/rust-space/ormponder/node_modules/viem/utils/promise/withRetry.ts:33:9)
9:47:01 AM WARN  realtime   Failed to fetch latest 'darwinia' block with error: TimeoutError: The request took too long to respond.

The first idea that comes to my mind is that the node rpc is down, no response to the eth_getBlockByNumber request. So, I write a script to test it by call the rpc method directly, the result shows that the rpc method is alive.

  • Test script:
import { ethers } from "ethers";

const providerUrl = "http://127.0.0.1:9944";

async function main() {
    for (let i = 0; i < 1000; i++) {
        const provider = new ethers.JsonRpcProvider(providerUrl);
        const latestBlock = await provider.getBlock("latest", true);
        console.log(`Block ${latestBlock.number} fetched`);
        await sleep(500);
    }
}

function sleep(ms) {
    return new Promise((resolve) => {
      setTimeout(resolve, ms);
    });
  }

await main();
  • Test script output:
Block 2561662 fetched
Block 2561662 fetched
Block 2561662 fetched
Block 2561667 fetched
Block 2561667 fetched

The only log that is worth noting in the RPC server side is:

2024-04-26 09:49:02.186  WARN tokio-runtime-worker jsonrpsee_server::server: HTTP serve connection failed hyper::Error(IncompleteMessage)

I have no idea what caused it. I would appreciate it if anyone has any insight into this?

@0xOlias
Copy link
Collaborator

0xOlias commented Apr 26, 2024

Thanks for opening an issue.

Could you try writing a similar script to the one here, but using viem (https://viem.sh)? Ponder uses viem internally to communicate with RPCs. If the request works with viem but not Ponder, it would be a clue.

@boundless-forest
Copy link
Author

Testing with viem script:

// 1. Import modules.
import { createPublicClient, http } from 'viem'
import { mainnet } from 'viem/chains'

const client = createPublicClient({
  chain: mainnet,
  transport: http('http://127.0.0.1:9944'),
})

async function main() {
    for (let i = 0; i < 100; i++) {
      const block = await client.request({
        method: 'eth_getBlockByNumber',
        params: ['latest', true],
      })
    
      console.log("fetched block: ", block.number)
      await sleep(500);
    }
}

function sleep(ms) {
    return new Promise((resolve) => {
      setTimeout(resolve, ms);
    });
}

await main();

The output:

fetched block:  0x271777
fetched block:  0x271779
fetched block:  0x271779
fetched block:  0x27177a
fetched block:  0x27177a

It also works with viem. Or should I tweak the script to test it again?

@0xOlias
Copy link
Collaborator

0xOlias commented Apr 26, 2024

Thanks. Could you share the "network" configuration you're using in ponder.config.ts?

@lyoungblood
Copy link

We are also seeing this issue as well in 0.4.9, and it seems to be preventing the Ponder indexer from becoming healthy:

at withRetry.delay.count.count (file:///app/node_modules/viem/utils/buildRequest.ts:152:21)

at processTicksAndRejections (node:internal/process/task_queues:95:5)

at attemptRetry (file:///app/node_modules/viem/utils/promise/withRetry.ts:37:22)

2:43:05 AM WARN  realtime   Failed to process 'base' block 13655019 with error: InvalidInputRpcError: Missing or invalid parameters.

Double check you have provided the correct parameters.

URL: https://base-mainnet.blastapi.io/<snip>

Request body: {"method":"eth_getLogs","params":[{"blockHash":"0xa73c5268d7ef48a8eb856066197ab5ccbb6020361de5264529007c8ea3d321ec","topics":[["0x0559884fd3a460db3073b7fc896cc77986f16e378210ded43186175bf646fc5f","0xcf583bb0c569eb967f806b11601c4cb93c10310485c67add5f8362c2f212321f","0x70483e6592cd5182d45ac970e05bc62cdcc90e9d8ef2c2dbe686cf383bcd7fc5","0xd52b2b9b7e9ee655fcb95d2e5b9e0c9f69e7ef2b8e9d2d0ea78402d576d22e22","0x13ed6866d4e1ee6da46f845c46d7e54120883d75c5ea9a2dacc1c4ca8984ab80","0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"]]}]}

Details: unknown block

Version: viem@1.21.3

@boundless-forest
Copy link
Author

boundless-forest commented Apr 26, 2024

  • ponder.config.ts:
const MAX_REQUESTS_PER_SECOND = 6;
export default createConfig({
  networks: {
    darwinia: {
      chainId: 46,
	  // transport: http("http://c1.darwinia-rpc.itering.io:9944/"), 
      transport: http("http://127.0.0.1:9944"), 
      maxRequestsPerSecond: MAX_REQUESTS_PER_SECOND,
    },
  },

To make things clear, I have omitted other networks.

@jaylmiller
Copy link
Collaborator

@boundless-forest thanks for the config. the server you provided in the config appears to be offline though.

based on your initial error message it looks like your server took too long to respond and our transport killed the request, causing the hyper::Error(IncompleteMessage) error on the server.

@boundless-forest
Copy link
Author

Sorry for the inconvenience. The service http://c1.darwinia-rpc.itering.io:9944/ is currently unavailable. I'm testing with a localhost node.

it looks like your server took too long to respond and our transport killed the request

Extract! I also guess this is the reason. But I'm confused why calling the same rpc method, ponder will trigger this error while using the test script above will not. Is there any limit or config in the ponder to monitor the request time? such as if the request doesn't get a server response in xxx seconds, it will be killed. If there is, I can tweak the value and try again.

@jaylmiller
Copy link
Collaborator

jaylmiller commented Apr 28, 2024

@boundless-forest Yes. It looks like your using the http viem transport based on the network config you sent. That function takes a second optional configuration object, which has an option for setting the timeout:

type HttpTransportConfig = {
  /**
   * Whether to enable Batch JSON-RPC.
   * @link https://www.jsonrpc.org/specification#batch
   */
  batch?: boolean | BatchOptions
  /**
   * Request configuration to pass to `fetch`.
   * @link https://developer.mozilla.org/en-US/docs/Web/API/fetch
   */
  fetchOptions?: HttpOptions['fetchOptions']
  /** The key of the HTTP transport. */
  key?: TransportConfig['key']
  /** The name of the HTTP transport. */
  name?: TransportConfig['name']
  /** The max number of times to retry. */
  retryCount?: TransportConfig['retryCount']
  /** The base delay (in ms) between retries. */
  retryDelay?: TransportConfig['retryDelay']
  /** The timeout (in ms) for the HTTP request. Default: 10_000 */
  timeout?: TransportConfig['timeout']
}

So you can try changing your network config:

const TIMEOUT_MS  = 60_000;

export default createConfig({
  networks: {
    darwinia: {
      chainId: 46,
	  // transport: http("http://c1.darwinia-rpc.itering.io:9944/"), 
      transport: http("http://127.0.0.1:9944", {timeout: TIMEOUT_MS}), 
      maxRequestsPerSecond: MAX_REQUESTS_PER_SECOND,
    },
  },

@github-project-automation github-project-automation bot moved this from Todo to Done in Ponder Roadmap May 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants