Add support for Otterscan JSON-RPC API extensions

[ZePedroResende]

Describe the feature

Otterscan is an Ethereum block explorer designed to be run locally with an archive node companion.
It's more private compared to other solutions because it allows you to query your node, avoiding sharing sensitive data with third parties.
This is achieved without an external indexer through custom RPC methods that currently are only available in Erigon.
The addition of this extension (even if behind a feature flag) would allow the community to run Otterscan with a different client other than Erigon.
Users would have access to useful information not available on the standard RPC endpoints, without an external indexer, which can be very helpful for data-intensive applications.

Endpoint checklist

Beta Give feedback

1. ots_getApiLevel
2. ots_getInternalOperations
3. ots_hasCode
4. ots_getTransactionError
5. ots_traceTransaction
6. ots_getBlockDetails
7. ots_getBlockDetailsByHash
8. ots_getBlockTransactions
9. ots_searchTransactionsBefore
10. ots_searchTransactionsAfter
11. ots_getTransactionBySenderAndNonce
12. ots_getContractCreator

Additional context

No response

[naps62]

@gakonst as discussed, can we get this assigned to @ZePedroResende ? I'll be working on this with him

I was thinking a good first step would be to get an initial endpoint done as a draft, mainly to start discussing architecture. I imagine some input will be required, so I want to optimize everyone's time :)

[gakonst]

SGTM! Let us know how to proceed here.

[mattsse]

cool, some pointers:
the way it's structure is that we define a trait per namespace with all the functions, like:

reth/crates/rpc/rpc-api/src/web3.rs

Lines 4 to 16 in 94ba83f

	/// Web3 rpc interface.
	#[cfg_attr(not(feature = "client"), rpc(server, namespace = "web3"))]
	#[cfg_attr(feature = "client", rpc(server, client, namespace = "web3"))]
	#[async_trait::async_trait]
	pub trait Web3Api {
	/// Returns current client version.
	#[method(name = "clientVersion")]
	async fn client_version(&self) -> RpcResult<String>;
	/// Returns sha3 of the given data.
	#[method(name = "sha3")]
	fn sha3(&self, input: Bytes) -> RpcResult<H256>;
	}

and then add a type that implements this like:

reth/crates/rpc/rpc/src/web3.rs

Lines 23 to 38 in 94ba83f

	#[async_trait]
	impl<N> Web3ApiServer for Web3Api<N>
	where
	N: NetworkInfo + 'static,
	{
	/// Handler for `web3_clientVersion`
	async fn client_version(&self) -> RpcResult<String> {
	let status = self.network.network_status().await.to_rpc_result()?;
	Ok(status.client_version)
	}
	/// Handler for `web3_sha3`
	fn sha3(&self, input: Bytes) -> RpcResult<H256> {
	Ok(keccak256(input))
	}
	}

[naps62]

@mattsse thanks, that helped a lot already

what's your preference regarding how to manage the work here? a single large branch where we built the feature and merge as a whole? merging small incremental changes, disabled through a feature flag?

[mattsse]

incremental would be ideal, for example

1. trait bindings
2. impl (can be done in multiple steps)
3. integration in rpc-builder+cli

[naps62]

Bringing back @gakonst 's comment from the PR, feels more on-topic here:

My main Q/concern here is: What new tables are needed to support these APIs?
Could we scope them very clearly, as well as understand 1) what stages (if any) we need to add to generate these new tables, 2) how much DB overhead they incur?
My main Q/concern here is: What new tables are needed to support these APIs?

makes sense. We went this route because of mattsse's suggestion, but I do admit we still need to learn more about that. also because we're still learning the codebase here, so any input is much appreciated

I was getting more familiar with reth storage because of this topic yesterday. Here's my thoughts so far (I'll let @ZePedroResende correct/complete me):

• ots_getTransactionBySenderAndNonce seems to need a (Address, u64) | H256 table
• ots_getContractCreator should need an Address | Address mapping
• ots_searchTransactionsBefore/After should be by far the biggest one: it requires mapping each address to any tx that ever touched it (outbound, inbound, internal).
• ots_getBlockDetails shouldn't need extra tables, but it does have an issuance object. in Ethereum this is now meaningless, but depending on what other chains are need to support, may need some computation
• ots_getTransactionError and ots_traceTransaction seem like they reuse the existing tracing code?

Maybe some of these can leverage existing tables that I overlooked?
I also assume we'll at the very least want to feature-gate these features to avoid the extra overhead unless explicitly needed?

[gakonst]

I also assume we'll at the very least want to feature-gate these features to avoid the extra overhead unless explicitly needed?

100% - for sure.

cc @joshieDo as these are all indices and it might overlap with the future etl work

[onbjerg]

@ZePedroResende moved the tasklist to the main issue

[0xAlcibiades]

This still seems to be missing a few parts including the erigon namespace for otterscan support to work. I see this was recently added to foundry and wonder if there'd be interest to bring things up to speed on this side @Rjected @Evalir @ZePedroResende

[0xAlcibiades]

Relates to: foundry-rs/foundry#5414