Teku Validator client unable to find keys #174

GLCNI · 2022-11-22T16:36:03Z

Describe the bug
Teku Validator client unable to find validator keys

To Reproduce
Start Sedge with network Gnosis and Teku for CL client. once synced start the validator.

had sedge started up a day ago, EL and CL clients synced okay. prompted to start the validator (at this point did not generate keys yet)
Not able to access the 'validator-data' folder until I changed permissions

keys generated restart validator client
sedge keys --network gnosis
places keys in ./keystore/validator_keys and password.txt in ./keystore

Teku is looking for keys in --validator-keys=/data/keys:/data/passwords
I think Teku requires password file per validator with same name structure but .txt extension

Tried to Import validators
ran the container to import the keys, and restarted the validator client
same error and looks like the validators are imported to
./validator-data/validators

The text was updated successfully, but these errors were encountered:

stdevMac · 2022-11-23T13:08:50Z

Hello @GLCNI, we are looking into your issue now!

Could you please provide us the docker-compose.yml file under the docker-compose-scripts folder?

GLCNI · 2022-11-23T13:34:52Z

docker-compose.yml

version: "3.9"
services:
  execution:
    stop_grace_period: 30s
    container_name: execution-client
    restart: unless-stopped
    image: ${EC_IMAGE_VERSION}
    networks:
    - sedge
    volumes:
    - ${EC_DATA_DIR}:/nethermind/data
    - ${EC_JWT_SECRET_PATH}:/tmp/jwt/jwtsecret
    ports:
    - 30303:30303/tcp
    - 30303:30303/udp
    expose:
    - 8545
    - 8551
    command:
    - --config=${EL_NETWORK}
    - --datadir=/nethermind/data
    - --log=${NETHERMIND_LOG_LEVEL}
    - --Sync.SnapSync=${EC_SNAP_SYNC_ENABLED}
    - --JsonRpc.Enabled=true
    - --JsonRpc.Host=0.0.0.0
    - --JsonRpc.Port=8545
    - --JsonRpc.EnabledModules=${EC_ENABLED_MODULES}
    - --JsonRpc.JwtSecretFile=/tmp/jwt/jwtsecret
    - --JsonRpc.EngineHost=0.0.0.0
    - --JsonRpc.EnginePort=8551
    - --Network.DiscoveryPort=30303
    - --HealthChecks.Enabled=${NETHERMIND_HEALTH_CHECKS_ENABLED}
    - --Pruning.CacheMb=${NETHERMIND_PRUNING_CACHEMB}
    logging:
      driver: json-file
      options:
        max-size: 10m
        max-file: "10"
  consensus:
    stop_grace_period: 30s
    container_name: consensus-client
    restart: unless-stopped
    image: ${CC_IMAGE_VERSION}
    networks:
    - sedge
    volumes:
    - ${CC_DATA_DIR}:/var/lib/teku
    - ${CC_JWT_SECRET_PATH}:/tmp/jwt/jwtsecret
    ports:
    - 9000:9000/tcp
    - 9000:9000/udp
    - 5054:5054/tcp
    expose:
    - 4000
    command:
    - --log-destination=CONSOLE
    - --logging=${CC_LOG_LEVEL}
    - --network=${CL_NETWORK}
    - --p2p-discovery-bootnodes=${CL_BOOTNODES}
    - --p2p-enabled=true
    - --p2p-port=9000
    - --p2p-peer-upper-bound=${CC_PEER_COUNT}
    - --rest-api-enabled=true
    - --rest-api-host-allowlist=*
    - --rest-api-interface=0.0.0.0
    - --rest-api-port=4000
    - --rest-api-cors-origins=*
    - --rest-api-docs-enabled=false
    - --data-beacon-path=/var/lib/teku/beacon
    - --data-path=/var/lib/teku
    - --data-storage-archive-frequency=2048
    - --data-storage-mode=PRUNE
    - --data-storage-non-canonical-blocks-enabled=false
    - --ee-endpoint=${EC_AUTH_URL}
    - --validators-proposer-default-fee-recipient=${CL_FEE_RECIPIENT}
    - --ee-jwt-secret-file=/tmp/jwt/jwtsecret
    - --metrics-enabled=true
    - --metrics-host-allowlist=*
    - --metrics-interface=0.0.0.0
    - --metrics-port=5054
    - --initial-state=${CHECKPOINT_SYNC_URL}/eth/v2/debug/beacon/states/finalized
    logging:
      driver: json-file
      options:
        max-size: 10m
        max-file: "10"
  validator-import:
    container_name: validator-import-client
    build:
      context: github.com/NethermindEth/teku-init-validator
    volumes:
    - ${VL_DATA_DIR}:/data
    - ${KEYSTORE_DIR}:/keystore
  validator:
    container_name: validator-client
    image: ${VL_IMAGE_VERSION}
    depends_on:
      validator-import:
        condition: service_completed_successfully
    networks:
    - sedge
    ports:
    - 5056:5056
    volumes:
    - ${VL_DATA_DIR}:/data
    command:
    - validator-client
    - --beacon-node-api-endpoint=${CC_API_URL}
    - --data-path=/data
    - --log-destination=CONSOLE
    - --validators-graffiti=${GRAFFITI}
    - --validator-keys=/data/keys:/data/passwords
    - --metrics-enabled=true
    - --metrics-host-allowlist=*
    - --metrics-interface=0.0.0.0
    - --metrics-port=5056
    - --validators-proposer-default-fee-recipient=${CL_FEE_RECIPIENT}
    logging:
      driver: json-file
      options:
        max-size: 10m
        max-file: "10"
networks:
  sedge:
    name: sedge_network

.env file

# --- Global configuration ---
EL_NETWORK=xdai
CL_NETWORK=gnosis
# --- Execution Layer - Execution Node - configuration ---
EC_IMAGE_VERSION=nethermind/nethermind:1.14.3
NETHERMIND_LOG_LEVEL=INFO
EC_ENABLED_MODULES=[Web3,Eth,Subscribe,Net,]
EC_METRICS_ENABLED=false
EC_NODENAME=Nethermind
NETHERMIND_METRICS_PUSH_GATEWAY_URL=http://localhost:9090/metrics
NETHERMIND_HEALTH_CHECKS_ENABLED=false
NETHERMIND_PRUNING_CACHEMB=2048
EC_DATA_DIR=./execution-data
EC_SNAP_SYNC_ENABLED=false
EC_JWT_SECRET_PATH=/home/glc/docker-compose-scripts/jwtsecret
# --- Consensus Layer - Beacon Node - configuration ---
CC_PEER_COUNT=50
CC_LOG_LEVEL=info
EC_API_URL=http://execution:8545
EC_AUTH_URL=http://execution:8551
CC_INSTANCE_NAME=Teku
CC_IMAGE_VERSION=consensys/teku:22.9.0
CC_DATA_DIR=./consensus-data
CC_JWT_SECRET_PATH=/home/glc/docker-compose-scripts/jwtsecret
CL_FEE_RECIPIENT=< >
CL_BOOTNODES="enr:-Iq4QMCTfIMXnow27baRUb35Q8iiFHSIDBJh6hQM5Axohhf4b6Kr_cOCu0htQ5WvVqKvFgY28893DHAg8gnBAXsAVqmGAX53x8JggmlkgnY0gmlwhLKAlv6Jc2VjcDI1NmsxoQK6S-Cii_KmfFdUJL2TANL3ksaKUnNXvTCv1tLwXs0QgIN1ZHCC>
CHECKPOINT_SYNC_URL=https://rpc-gbc.gnosischain.com
# --- Consensus Layer - Validator Node - configuration ---
CC_API_URL=http://consensus:4000
GRAFFITI=nethermind-teku
VL_LOG_LEVEL=info
VL_INSTANCE_NAME=TekuValidator
VL_IMAGE_VERSION=consensys/teku:22.9.0
KEYSTORE_DIR=./keystore
VL_DATA_DIR=./validator-data

stdevMac · 2022-11-23T15:15:30Z

Thanks!

Can you please run this command:

ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/   /' -e 's/-/|/'

and let me know if the output is the same that the one above:

   .
   |-consensus-data
   |---beacon
   |-----db
   |-----kvstore
   |---validator
   |-execution-data
   |---keystore
   |---logs
   |---nethermind_db
   |-----xdai
   |-------blockInfos
   |-------blocks
   |-------bloom
   |-------canonicalHashTrie
   |-------code
   |-------discoveryNodes
   |-------headers
   |-------metadata
   |-------peers
   |-------receipts
   |-------state
   |---------0
   |-------witness
   |-keystore
   |---validator_keys
   |-validator-data
   |---keys
   |---passwords
   |---validator

I have been trying in different moments, and for example, I tried to run the validator by hand without waiting to sync and it ran without issues, with v0.4.0.

I follow the next commands:

Run sedge keys --network gnosis
Run sedge cli --network gnosis -c teku -v teku

On the process of sync, I wait until I reached a point like this:

2022-11-23 14:41:36 -- [INFO] [GetRequest] Retrying request
2022-11-23 14:41:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.3:4000 is syncing
2022-11-23 14:41:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of execution nodes...
2022-11-23 14:41:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.4:8545 is syncing
2022-11-23 14:46:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of consensus nodes...
2022-11-23 14:46:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.3:4000 is synced
2022-11-23 14:46:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of execution nodes...
2022-11-23 14:46:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.4:8545 is syncing
2022-11-23 14:51:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of consensus nodes...
2022-11-23 14:51:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.3:4000 is synced
2022-11-23 14:51:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of execution nodes...
2022-11-23 14:51:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.4:8545 is syncing
2022-11-23 14:56:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of consensus nodes...
2022-11-23 14:56:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.3:4000 is synced
2022-11-23 14:56:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of execution nodes...
2022-11-23 14:56:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.4:8545 is syncing
2022-11-23 15:01:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of consensus nodes...
2022-11-23 15:01:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.3:4000 is synced
2022-11-23 15:01:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of execution nodes...
2022-11-23 15:01:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.4:8545 is syncing
2022-11-23 15:06:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of consensus nodes...
2022-11-23 15:06:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.3:4000 is synced
2022-11-23 15:06:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of execution nodes...
2022-11-23 15:06:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.4:8545 is syncing
2022-11-23 15:11:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of consensus nodes...
2022-11-23 15:11:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.3:4000 is synced
2022-11-23 15:11:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of execution nodes...
2022-11-23 15:11:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.4:8545 is syncing

That means that both nodes, execution, and consensus are syncing, open another terminal and:
3. Run cd docker-compose-scripts and docker compose up validator, and this is the output

Attaching to validator-client
validator-client  | 2022-11-23 15:05:34,344 main INFO Configuring logging for destination: console
validator-client  | 2022-11-23 15:05:34,404 main INFO Logging includes events: true
validator-client  | 2022-11-23 15:05:34,406 main INFO Logging includes validator duties: true
validator-client  | 2022-11-23 15:05:34,406 main INFO Logging includes color: true
validator-client  | 2022-11-23 15:05:34,501 main INFO Include P2P warnings set to: false
validator-client  | 2022-11-23 15:05:37.494 INFO  - Ignoring non-configurable constants supplied in network configuration: BLS_WITHDRAWAL_PREFIX, TARGET_AGGREGATORS_PER_COMMITTEE, RANDOM_SUBNETS_PER_VALIDATOR, EPOCHS_PER_RANDOM_SUBNET_SUBSCRIPTION, DOMAIN_BEACON_PROPOSER, DOMAIN_BEACON_ATTESTER, DOMAIN_RANDAO, DOMAIN_DEPOSIT, DOMAIN_VOLUNTARY_EXIT, DOMAIN_SELECTION_PROOF, DOMAIN_AGGREGATE_AND_PROOF, DOMAIN_SYNC_COMMITTEE, DOMAIN_CONTRIBUTION_AND_PROOF, TARGET_AGGREGATORS_PER_SYNC_SUBCOMMITTEE, DOMAIN_SYNC_COMMITTEE_SELECTION_PROOF, SYNC_COMMITTEE_SUBNET_COUNT
validator-client  | 2022-11-23 15:05:37.527 INFO  - Ignoring unknown items in network configuration: DOMAIN_APPLICATION_BUILDER
validator-client  | 2022-11-23 15:05:39.889 INFO  - Teku version: teku/v22.9.0/linux-x86_64/-eclipseadoptium-openjdk64bitservervm-java-17
validator-client  | 2022-11-23 15:05:39.901 INFO  - This software is licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
validator-client  | 2022-11-23 15:05:40.485 INFO  - validator-api-enabled is false, not starting rest api.
validator-client  | 2022-11-23 15:05:40.505 INFO  - Storing validator data in: /data/validator
validator-client  | 2022-11-23 15:05:40.540 INFO  - Using optimized BLST library
validator-client  | 2022-11-23 15:05:40.586 INFO  - Starting metrics http service on 0.0.0.0:5056
validator-client  | 2022-11-23 15:05:41.019 INFO  - BLS: loaded BLST library
validator-client  | 2022-11-23 15:05:41.112 INFO  - Loading 5 validator keys...
validator-client  | 2022-11-23 15:05:41.737 INFO  - Metrics service started and listening on 0.0.0.0:5056
validator-client  | 2022-11-23 15:05:46.053 INFO  - Loaded 5 Validators: b7bdb9b, 8db4e97, 8292e87, a35007c, 885583c
validator-client  | 2022-11-23 15:05:46.270 INFO  - Starting EventSource client using URI: http://consensus:4000/eth/v1/events?topics=head
validator-client  | 2022-11-23 15:05:46.320 INFO  - Connected to EventSource stream.
validator-client  | 2022-11-23 15:05:46.346 INFO  - Validator   *** Successfully connected to beacon node event stream
validator-client  | 2022-11-23 15:05:46.506 WARN  - Unable to retrieve status for validator b7bdb9b.
validator-client  | 2022-11-23 15:05:46.506 WARN  - Unable to retrieve status for validator 8db4e97.
validator-client  | 2022-11-23 15:05:46.507 WARN  - Unable to retrieve status for validator 8292e87.
validator-client  | 2022-11-23 15:05:46.507 WARN  - Unable to retrieve status for validator a35007c.
validator-client  | 2022-11-23 15:05:46.508 WARN  - Unable to retrieve status for validator 885583c.
validator-client  | 2022-11-23 15:05:48.523 INFO  - Not performing attestation duties for slot 6044521 because epoch 377782 is too far ahead of the current epoch UNDEFINED
validator-client  | 2022-11-23 15:05:50.151 WARN  - Validator   *** Slashing protection not loaded for validators: b7bdb9b, 885583c, 8db4e97, a35007c, 8292e87
^CGracefully stopping... (press Ctrl+C again to force)

Will try your compose, check the differences, and see what is the source of the problem. WIll keep you updated, Thanks for letting us know!

GLCNI · 2022-11-23T16:24:39Z

Ran the command, this is the output

|-consensus-data
   |---beacon
   |-----db
   |-----kvstore
   |---validator
   |-execution-data
   |---keystore
   |---logs
   |---nethermind_db
   |-----xdai
   |-------blockInfos
   |-------blocks
   |-------bloom
   |-------canonicalHashTrie
   |-------code
   |-------discoveryNodes
   |-------headers
   |-------metadata
   |-------peers
   |-------receipts
   |-------state
   |---------0
   |-------witness
   |-keystore
   |---validator_keys
   |-validator-data
   |---validator
   |---validators   
   |-----0x81854d896dd31edc5fe805227a607d5cbf09092f14f24e7054c13f9068116ce86a8607b59572d6dfd65d59ff8ac23c65
   |-----0x87325850383845784da9e02a6b660378f106ca58ae7f511a5fe5508f12928102b605ded3b4efc5901b4cc9854fe70894
   |-----0xa3c65ebf5dba6f1e2b1f9fe4860ca0ac937dc15dc8d501d000743f16bf02b9e18df5da98b5302ac092678044e5cd42af
   |-----0xb1bbc475c1db718e40e7a15c3da2778dcc96602d92cc0a78232b11ca8d6d4d9eeb62d58cb3a1f1f05f4facf6a10857ec

stdevMac · 2022-11-23T17:48:16Z

Can you please provide me the args to sedge cli you are using? because the compose that is generated is different from the one I am generating locally, and I am using basic config.

stdevMac · 2022-11-24T00:06:19Z

Hey @GLCNI I have been testing, a few things, and all of them worked for me, here is what I did overall:

Generate keys for gnosis using sedge keys --network gnosis
Run sedge cli only to generate the compose, without running it. sedge cli --network gnosis -c teku
Deleted my compose, and copy the one provided by you
Deleted my .env file and copy the one you provided, just modifying the path for my local machine, changing only the usernames.
Run these commands on this order:

docker compose up execution -d
docker compose up execution -d
docker compose up validator-import
dicker compose up validator

And everything works.

I am running it on a Ubuntu machine.

Can you please run the docker compose with that .env in a new folder, generated using sedge cli? or cleaning the validator data or, the sedge folder. After that, run the validator-import container docker compose validator-import that will move for you all the keystores to the right location for teku, and prepare the environment.

After that, run the validator and please, let us know if the issue persists.

Thanks in advance!

GLCNI · 2022-11-25T15:53:29Z

Solved: @stdevMac deleted the validator data folder and re-imported validator keys,

Problem:
it was using a cached image for validator import from a previous build, likely lighthouse so had to run
docker system prune -a before this would work. because I've been testing with different clients I had this issue, but this might be an issue when switching clients.

github.com/NethermindEth/teku-init-validator
which works fine, keys detected in teku validator client

stdevMac · 2022-11-25T16:35:57Z

Thanks @GLCNI for letting us know! If you need anything else, please feel free to submit an issue at any time 🙌

AntiD2ta assigned stdevMac Nov 23, 2022

AntiD2ta added the bug Something isn't working label Nov 23, 2022

stdevMac closed this as completed Nov 25, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Teku Validator client unable to find keys #174

Teku Validator client unable to find keys #174

GLCNI commented Nov 22, 2022

stdevMac commented Nov 23, 2022

GLCNI commented Nov 23, 2022

stdevMac commented Nov 23, 2022

GLCNI commented Nov 23, 2022

stdevMac commented Nov 23, 2022

stdevMac commented Nov 24, 2022

GLCNI commented Nov 25, 2022

stdevMac commented Nov 25, 2022

Teku Validator client unable to find keys #174

Teku Validator client unable to find keys #174

Comments

GLCNI commented Nov 22, 2022

stdevMac commented Nov 23, 2022

GLCNI commented Nov 23, 2022

stdevMac commented Nov 23, 2022

GLCNI commented Nov 23, 2022

stdevMac commented Nov 23, 2022

stdevMac commented Nov 24, 2022

GLCNI commented Nov 25, 2022

stdevMac commented Nov 25, 2022