ListValidatorBalances fails for multiple epochs

[alrevuelta]

🐞 Bug Report

Description

The gRPC call ListValidatorBalances fails for multiple epochs (i.e. 11000, 40000, 70000) but works fine for others (i.e. 5000, 50000 and recent ones close to head). Note that there is no connection with the Indices, so regardless of the indice the call fails.

Has this worked before in a previous version?

Can't tell

🔬 Minimal Reproduction

• Set up a prysm beacon-node v2.0.2 running in localhost:4000 and wait until is synched.
• Note that the beacon-node is NOT running in archival mode, so slots-per-archive-point has its default value.
• Run the following code snippet with epoch=11000, epoch=40000 or epoch=70000. Note that for example epoch=5000 works as expected.

package main

import (
	"context"
	ethTypes "github.com/prysmaticlabs/eth2-types"
	ethpb "github.com/prysmaticlabs/prysm/v2/proto/prysm/v1alpha1"
	log "github.com/sirupsen/logrus"
	"google.golang.org/grpc"
	"time"
)

func main() {
	dialContext, err := grpc.DialContext(context.Background(), "localhost:4000", grpc.WithInsecure())
	if err != nil {
		log.Fatal("error: ", err)
	}

	beaconClient := ethpb.NewBeaconChainClient(dialContext)

	epoch := 70000
	validatorIndex := 7

	request := ethpb.ListValidatorBalancesRequest{
		QueryFilter: &ethpb.ListValidatorBalancesRequest_Epoch{ethTypes.Epoch(epoch)},
		Indices:     []ethTypes.ValidatorIndex{ethTypes.ValidatorIndex(uint64(validatorIndex))},
	}

	ctx, cancel := context.WithTimeout(context.Background(), 120*time.Second)
	defer cancel()

	resp, err := beaconClient.ListValidatorBalances(ctx, &request)
	if err != nil {
		log.Fatal(err)
	}

	log.Info("Resp: ", resp)
}

🔥 Error

The script above returns the following error:

FATA[0030] rpc error: code = Internal desc = Could not get state: could not process block: could not process block operation: could not process block attestations: could not verify attestation at index 0 in block: could not interface with committee cache: context deadline exceeded.

Note that I can't see any error message in the beacon chain, just that there is an incoming connexion. Note that I'm using info as verbosity level.

│ time="2021-12-02 18:37:10" level=info msg="New gRPC client connected to beacon node" addr="127.0.0.1:48726" prefix=rpc │

🌍 Your Environment

Operating System:

Prysm beacon-node runs in Linux, script in macOS.

What version of Prysm are you running? (Which release)

v2.0.2

Anything else relevant (validator index / public key)?

NA

[prestonvanloon]

We removed the timeout in #9957. So this should no longer be an issue in the next release. The default value of archival points can often exceed 30s for certain requests of old states.

Closing for now as #9957 will resolve your issue. This hasn't landed in a release yet, but you can run from the develop branch or commit 7cbef10.