Cannot find custody address when submitting message to local hub instance

[michaelhly]

What is the bug?
Started a local instance of hubble and tried to submit messages with script, running into issues with this validation:
https://github.com/farcasterxyz/hub-monorepo/blob/main/apps/hubble/src/storage/engine/index.ts#L966-L968

I see this event log from hubble in my console:

farsentimiento-hubble-1  | {"level":30,"time":1691751646306,"pid":1,"hostname":"a27a59cfc2f5","component":"EthEventsProvider","event":{"to":"0xd51C731A9e5b2b79Fe6E91a2451d7CC6DCb25DF6","id":"14103","blockNumber":9222145},"msg":"cacheIdRegistryEvent: fid 14103 assigned to 0xd51C731A9e5b2b79Fe6E91a2451d7CC6DCb25DF6 in block 9222145"}

And running the following code inside a REPL yields the same address:

from eth_account import Account

Account.enable_unaudited_hdwallet_features()
signer = Account.from_mnemonic(FARCASTER_MNEMONIC)
print(signer.address)
# 0xd51C731A9e5b2b79Fe6E91a2451d7CC6DCb25DF6

However, when submitting a message to my hub instance I get this validation error:

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.INVALID_ARGUMENT
	details = "unknown fid: 14103"
	debug_error_string = "UNKNOWN:Error received from peer ipv6:%5B::1%5D:2283 {grpc_message:"unknown fid: 14103", grpc_status:3, created_time:"2023-08-11T19:01:09.804192+08:00"}"
>

How can it be reproduced? (optional)

1. Run docker-compose up with the docker-compose file here to start a hub:
   https://github.com/michaelhly/FarGlot/blob/master/docker-compose.yml

2. Run the following script to submit messages to a local hub instance:
   https://github.com/michaelhly/FarGlot/blob/master/scripts/boostrap_hub_with_recent_casts.py

Additional context (optional)
You need to install the following python dependencies to run the script:

python = ">=3.9, <3.11"
PyNaCl = "^1.5.0"
eth-account = "^0.9.0"
blake3 = "^0.3.3"
protobuf = "4.22.1"
requests = "^2.31.0"
urllib3 = "1.26.6"
grpcio = "1.53.0"
grpcio-tools = "1.53.0"

[sanjayprabhu]

cacheIdRegistryEvent means that it was only cached, the hub waits for 6 confirmations before actually storing the event. You need to wait for the submitIdRegistryEvent success log line generated within https://github.com/farcasterxyz/hub-monorepo/blob/main/apps/hubble/src/eth/ethEventsProvider.ts#L596.

Or if you want to do this programmatically, subscribe to the hub events and wait for the MergeIdRegistryEvent with the fid you're interested in before taking any action with the fid: https://github.com/farcasterxyz/hub-monorepo/blob/main/protobufs/schemas/hub_event.proto#L14

[michaelhly]

@sanjayprabhu can we change the custodyAddress look up to look up from the cacheIdRegistry if some env var is set for the sake of local development?

nvm, the cache key is blockNumber ...

[sanjayprabhu]

We have to go through the submit process, we cannot look up from the cache.

But, you could potentially change the number of confirmations to 1 or 2: https://github.com/farcasterxyz/hub-monorepo/blob/main/apps/hubble/src/eth/ethEventsProvider.ts#L69 (be sure to update eventPollingInterval as well so it doesn't go negative). 0 confirmations is untested, things may break.