Duplicated provider without any configuration

[maoueh]

Do you want to request a feature or report a bug?

I would say more a mis-behavior

What is the current behavior?

I was doing some testing with Arweave, in my config I have a single provider defined, but it seems graph-node is actually auto duplicating it to become like 20 providers, all with the same endpoint:

Jan 27 13:12:41.566 DEBG Cleaning up large notifications after about 300s, channel: chain_head_updates, component: ChainHeadUpdateListener > NotificationListener
Jan 27 13:12:41.714 INFO Connecting to Firehose to get chain identifier, provider: arweave-mainnet-0, provider: arweave-mainnet-0
Jan 27 13:12:41.714 INFO Requesting genesis block from firehose, provider: arweave-mainnet-0
Jan 27 13:12:41.715 DEBG Connecting to firehose to retrieve block for number 0, provider: arweave-mainnet-0
Jan 27 13:12:41.715 INFO Connecting to Firehose to get chain identifier, provider: arweave-mainnet-1, provider: arweave-mainnet-1
Jan 27 13:12:41.715 INFO Requesting genesis block from firehose, provider: arweave-mainnet-1
Jan 27 13:12:41.715 DEBG Connecting to firehose to retrieve block for number 0, provider: arweave-mainnet-1
...
Jan 27 13:12:41.717 DEBG Connecting to firehose to retrieve block for number 0, provider: arweave-mainnet-19
Jan 27 13:12:42.088 DEBG Retrieving block(s) from firehose, provider: arweave-mainnet-14
...
Jan 27 13:12:42.308 DEBG Retrieving block(s) from firehose, provider: arweave-mainnet-8
Jan 27 13:12:42.550 INFO Connected to Firehose, genesis_block: #900000 (d750464ec1fb149c031a4ed76a1b3bfc324f800b5b5ad2b66ade5906e8520ecd95e351a8b2fa1c673f83f0594775dabb), provider: arweave-mainnet-14, provider: arweave-mainnet-14
...
Jan 27 13:12:42.879 INFO Connected to Firehose, genesis_block: #900000 (d750464ec1fb149c031a4ed76a1b3bfc324f800b5b5ad2b66ade5906e8520ecd95e351a8b2fa1c673f83f0594775dabb), provider: arweave-mainnet-6, provider: arweave-mainnet-6

What is the expected behavior?

It seems really weird to see from an “operator” perspective, I imagine this is to properly to the load balancing and handling more than 100 streams which is a current limit.

However, it feels pretty weird, I would imagine there is a better way to implement to avoid "artificially" creating new provider out of the configured one. We should maybe even think about updating tonic library if we need to add feature to better to the load balancing.

• It should be configurable, by default we should scale to 1
• We should investigate how to convert that to a different way to perform the load balancing
• We should definitely avoid connecting to each of the "artificial" one, they all point to the same endpoint, so there should all return the same thing for the genesis block (don't bother about the 900000 value, it's a limitation of our endpoint)

[sduchesneau]

This burst of connections (multiplied by the scale) seems to cause more trouble: firehose server will trigger its rate-limiting mechanism when receiving a large burst of connection with status: Unavailable, expecting the client to back-off and retry, but the graph-node client then detects the whole provider as "broken", ignoring the chain.

DEBG Retrieving block(s) from firehose, provider: firehose-mainnet
ERRO Connection to provider failed. Not using this provider, error: firehose error status: Unavailable, message: "rate limit exceeded", details: [], metadata: MetadataMap { headers: {} }, provider: firehose-mainnet
ERRO No non-broken providers available for chain mainnet; ignoring this chain

[paymog]

I'm running into this issue as well. I did a bit of digging and found that we instantiate the connections here and it's based on this default config.

I haven't been able to find clear documentation on why/when the conn_pool_size should be changed. Are there docs on how graph node behaves when the pool size for a firehose provider is too low? What intuition/metrics should indexers use to properly configure this config value for firehose providers?

Seems like this might be related to #3879?

[leoyvens]

@paymog The rule of thumb is 1 connection per 100 subgraphs using the provider. So in most situations conn_pool_size can be set a lot lower than the default of 20.

[github-actions]

Looks like this issue has been open for 6 months with no activity. Is it still relevant? If not, please remember to close it.

[leoyvens]

I think #4778 fixed this.