How do I setup Full Archival Node for klaytn EN

[aki263]

Describe the bug
I am trying to setup full archival self hosted node for klaytn EN.
I have downloaded the snapshot and extracted it.

How to reproduce

Run docker:

docker run -d --name ken \
  --restart always \
  -p 0.0.0.0:8551:8551 \
  -v $CONF_DIR:/klaytn-docker-pkg/conf \
  -v $DATA_DIR:/var/kend/data \
  -v $LOGS_DIR:/var/kend/logs \
  klaytn/klaytn:latest \
  /bin/bash -c "kend start && touch /var/kend/logs/kend.out && tail -f /var/kend/logs/kend.out"

kend.conf

LOG_DIR=/var/kend/logs
DATA_DIR=/var/kend/data
NETWORK="cypress"
SERVER_TYPE="http"
SYNCMODE="full"
VERBOSITY=3
RPC_ENABLE=1
RPC_API="admin, debug, klay, miner, net, personal, rpc, txpool, web3"
RPC_ADDR="0.0.0.0"
RPC_VHOSTS="*"
METRICS=1
TXPOOL_EXEC_SLOTS_ALL=4096
TXPOOL_NONEXEC_SLOTS_ALL=4096
TXPOOL_EXEC_SLOTS_ACCOUNT=4096
TXPOOL_NONEXEC_SLOTS_ACCOUNT=4096
ADDITIONAL="--upstream-en https://archive-en.cypress.klaytn.net --gcmode=archive"

Now when I run

curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0", "method":"debug_traceBlockByNumber", "params":["0x6147031", {"disableStack": true, "disableMemory": true, "disableStorage": true, "tracer": "callTracer", "tracerConfig": {"withLog": true}}],"id":1,"jsonrpc":"2.0"}' localhost:8551

Response of Curl:
{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"historical state unavailable. tried regeneration but not possible, possibly due to state migration/pruning or global state saving interval is bigger than reexec value (reexec=128)"}}

INFO[05/17,16:08:41 Z] [4] Logging configured                        rotate=false format=terminal location=/var/kend/logs/kend.out
INFO[05/17,16:08:41 Z] [31] Enabling metrics collection
INFO[05/17,16:08:41 Z] [18] Setting connection type                   nodetype=en conntype=1
INFO[05/17,16:08:41 Z] [18] Cypress bootnodes are set
INFO[05/17,16:08:41 Z] [18] Bootnode - Add Seed                       Node="kni://18b36118cce093673499fc6e9aa196f047fe17a0de35b6f2a76a4557802f6abf9f89aa5e7330e93c9014b714b9df6378393611efe39aec9d3d831d6aa9d617ae@52.78.22.32:32323?ntype=bn"
INFO[05/17,16:08:41 Z] [18] Bootnode - Add Seed                       Node="kni://63f1c96874da85140ecca3ce24875cb5ef28fa228bc3572e16f690db4a48fc8067502d2f6e8f0c66fb558276a5ada1e4906852c7ae42b0003e9f9f25d1e123b1@52.194.200.217:32323?ntype=bn"
INFO[05/17,16:08:41 Z] [18] Bootnode - Add Seed                       Node="kni://94cc15e2014b86584908707de55800c0a2ea8a24dc5550dcb507043e4cf18ff04f21dc86ed17757dc63b1fa85bb418b901e5e24e4197ad4bbb0d96cd9389ed98@18.136.251.28:32323?ntype=bn"
INFO[05/17,16:08:41 Z] [18] Setting MaxPhysicalConnections            MaxPhysicalConnections=10
INFO[05/17,16:08:41 Z] [18] Cypress network ID is set                 networkid=8217
INFO[05/17,16:08:41 Z] [18] Raised fd limit to process's maximum value  fd=1048576
INFO[05/17,16:08:41 Z] [18] Cypress network ID is set                 networkid=8217
INFO[05/17,16:08:41 Z] [18] Archiving mode of this node               isArchiveMode=true
INFO[05/17,16:08:41 Z] [40] Ntp time check                            local=2024-05-17T16:08:41.932398264Z remote=2024-05-17T16:08:41.93113388Z
INFO[05/17,16:08:41 Z] [40] Starting peer-to-peer node                instance=Klaytn/v1.12.1/linux-amd64/go1.20.6
INFO[05/17,16:08:41 Z] [48] Non-single database is used for persistent storage  DBType=LevelDB
INFO[05/17,16:08:41 Z] [48] LevelDB configurations                    path=/var/kend/data/chaindata/misc levelDBCacheSize=16 openFilesLimit=10485 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[05/17,16:08:42 Z] [48] LevelDB configurations                    path=/var/kend/data/chaindata/header levelDBCacheSize=38 openFilesLimit=26214 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[05/17,16:08:42 Z] [48] LevelDB configurations                    path=/var/kend/data/chaindata/body   levelDBCacheSize=38 openFilesLimit=26214 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[05/17,16:08:43 Z] [48] LevelDB configurations                    path=/var/kend/data/chaindata/receipts levelDBCacheSize=38 openFilesLimit=26214 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[05/17,16:08:43 Z] [48] LevelDB configurations                    path=/var/kend/data/chaindata/statetrie_migrated_147954907/0 levelDBCacheSize=76 openFilesLimit=52428 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[05/17,16:08:44 Z] [48] LevelDB configurations                    path=/var/kend/data/chaindata/statetrie_migrated_147954907/1 levelDBCacheSize=76 openFilesLimit=52428 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[05/17,16:08:45 Z] [48] LevelDB configurations                    path=/var/kend/data/chaindata/statetrie_migrated_147954907/2 levelDBCacheSize=76 openFilesLimit=52428 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[05/17,16:08:45 Z] [48] LevelDB configurations                    path=/var/kend/data/chaindata/statetrie_migrated_147954907/3 levelDBCacheSize=76 openFilesLimit=52428 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[05/17,16:08:46 Z] [48] Created a sharded database                dbType=statetrie numShards=4
INFO[05/17,16:08:46 Z] [48] LevelDB configurations                    path=/var/kend/data/chaindata/txlookup                       levelDBCacheSize=16 openFilesLimit=10485 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[05/17,16:08:46 Z] [48] LevelDB configurations                    path=/var/kend/data/chaindata/bridgeservice                  levelDBCacheSize=16 openFilesLimit=5242  useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[05/17,16:08:46 Z] [48] LevelDB configurations                    path=/var/kend/data/chaindata/snapshot                       levelDBCacheSize=22 openFilesLimit=15728 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[05/17,16:08:46 Z] [30] Successfully loaded governance state from database  blockNumber=153778176
INFO[05/17,16:08:46 Z] [41] Initialised chain configuration           config="{ChainID: 8217 IstanbulCompatibleBlock: 86816005 LondonCompatibleBlock: 86816005 EthTxTypeCompatibleBlock: 86816005 MagmaCompatibleBlock: 99841497 KoreCompatibleBlock: 119750400 ShanghaiCompatibleBlock: 135456000 CancunCompatibleBlock: 147534000 RandaoCompatibleBlock: 147534000 KIP103CompatibleBlock: 119750400 KIP103ContractAddress 0xD5ad6D61Dd87EdabE2332607C328f5cc96aeCB95 SubGroupSize: 22 UnitPrice: 25000000000 DeriveShaImpl: 2 Engine: istanbul}"
INFO[05/17,16:08:46 Z] [41] Initialising Klaytn protocol              versions="[65 64]" network=8217
INFO[05/17,16:08:46 Z] [5] Loaded most recent local header           number=153779165 hash=d88b56…ed58b5 td=153779166 age=2d6h32m
INFO[05/17,16:08:46 Z] [5] Loaded most recent local full block       number=153779165 hash=d88b56…ed58b5 td=153779166 age=2d6h32m
INFO[05/17,16:08:46 Z] [5] Loaded most recent local fast block       number=153779165 hash=d88b56…ed58b5 td=153779166 age=2d6h32m
INFO[05/17,16:08:46 Z] [5] prefetchTxWorkers are started             num=32
INFO[05/17,16:08:46 Z] [41] Live pruning is disabled because flag not stored in database
INFO[05/17,16:08:46 Z] [5] InitDeriveSha                             initial=0 withGov=true
INFO[05/17,16:08:46 Z] [5] Loaded local transaction journal          transactions=0 dropped=0
INFO[05/17,16:08:46 Z] [5] Regenerated local transaction journal     transactions=0 accounts=0
INFO[05/17,16:08:47 Z] [40] Starting P2P networking
INFO[05/17,16:08:49 Z] [33] Create udp                                config="{NetworkID:8217 PrivateKey:<nil> AnnounceAddr:[::]:32323 NodeDBPath:/var/kend/data/nodes NetRestrict:<nil> Bootnodes:[kni://18b36118cce093673499fc6e9aa196f047fe17a0de35b6f2a76a4557802f6abf9f89aa5e7330e93c9014b714b9df6378393611efe39aec9d3d831d6aa9d617ae@52.78.22.32:32323?ntype=bn kni://63f1c96874da85140ecca3ce24875cb5ef28fa228bc3572e16f690db4a48fc8067502d2f6e8f0c66fb558276a5ada1e4906852c7ae42b0003e9f9f25d1e123b1@52.194.200.217:32323?ntype=bn kni://94cc15e2014b86584908707de55800c0a2ea8a24dc5550dcb507043e4cf18ff04f21dc86ed17757dc63b1fa85bb418b901e5e24e4197ad4bbb0d96cd9389ed98@18.136.251.28:32323?ntype=bn] Unhandled:<nil> Id:0b165dfb53bf5c92087687790560269fb947232b8c29eabb2b6197c78203b3fa66aa78b26d7245ec9d2fcbbba892416a66d0da4f3dc90c10220e4123d1bb1102 Addr:[::]:32323 udp:<nil> Conn:0xc025906538 NodeType:3 MaxNeighborsNode:0 AuthorizedNodes:[]}"
INFO[05/17,16:08:49 Z] [34] UDP listener up                           self="kni://0b165dfb53bf5c92087687790560269fb947232b8c29eabb2b6197c78203b3fa66aa78b26d7245ec9d2fcbbba892416a66d0da4f3dc90c10220e4123d1bb1102@[::]:32323?ntype=en"
INFO[05/17,16:08:49 Z] [40] Started P2P server                        id=0b165dfb53bf5c92 multichannel=false
INFO[05/17,16:08:49 Z] [40] IPC endpoint opened                       url=/var/kend/data/klay.ipc
INFO[05/17,16:08:49 Z] [40] HTTP endpoint opened                      url=http://0.0.0.0:8551     cors= vhosts=*
INFO[05/17,16:08:49 Z] [40] RLPx listener up                          self="kni://0b165dfb53bf5c92087687790560269fb947232b8c29eabb2b6197c78203b3fa66aa78b26d7245ec9d2fcbbba892416a66d0da4f3dc90c10220e4123d1bb1102@[::]:32323?ntype=en"
INFO[05/17,16:08:49 Z] [44] Start listening chain head event to update stakingInfoCache.
INFO[05/17,16:08:53 Z] [33] [Dial] Add dial candidate from static nodes  id=c378208d4d09e95a NodeType=2 ip=20.196.67.20 mainPort=32323 port=[32323]
INFO[05/17,16:08:53 Z] [33] [Dial] Add dial candidate from static nodes  id=f060f60df76db002 NodeType=2 ip=13.250.137.30 mainPort=32323 port=[32323]
INFO[05/17,16:08:53 Z] [33] Added a single channel P2P Peer           id=f060f60df76db002 conn=trusted-staticdial peerID=f060f60df76db002
INFO[05/17,16:08:53 Z] [33] Added a single channel P2P Peer           id=c378208d4d09e95a conn=trusted-staticdial peerID=c378208d4d09e95a
INFO[05/17,16:08:55 Z] [28] Block synchronisation started
INFO[05/17,16:08:55 Z] [51] Mining aborted due to sync
INFO[05/17,16:08:56 Z] [28] Downloader queue stats                    receiptTasks=0 blockTasks=60 stakingInfoTasks=0 itemSize=4.00kB throttle=8192
INFO[05/17,16:08:57 Z] [5] Inserted a new block                      number=153779166 hash=8fc29e…5ff206 txs=48 gas=3163406 elapsed=510.095ms processTxs=35.495ms finalize=17.102ms validateState=795.145µs totalWrite=16.415262ms trieWrite=16.064674ms
INFO[05/17,16:08:57 Z] [5] Inserted a new block                      number=153779167 hash=e3f261…b72bf3 txs=57 gas=3774035 elapsed=408.863ms processTxs=48.457ms finalize=26.425ms validateState=837.029µs totalWrite=13.724149ms trieWrite=13.367444ms
INFO[05/17,16:08:58 Z] [5] Inserted a new block                      number=153779168 hash=149c49…25e87d txs=53 gas=3940195 elapsed=425.283ms processTxs=38.266ms finalize=17.459ms validateState=1.161ms   totalWrite=47.135177ms trieWrite=46.752624ms
INFO[05/17,16:08:59 Z] [5] Inserted a new block                      number=153779169 hash=84e0a5…1777c4 txs=67 gas=6049008 elapsed=1.335s    processTxs=729.443ms finalize=18.566ms validateState=1.252ms   totalWrite=47.623597ms trieWrite=47.247879ms
INFO[05/17,16:08:59 Z] [5] Inserted a new block                      number=153779170 hash=85c77c…2bf06e txs=73 gas=5600698 elapsed=509.431ms processTxs=74.778ms  finalize=23.448ms validateState=1.539ms   totalWrite=18.309417ms trieWrite=17.916143ms
INFO[05/17,16:09:00 Z] [5] Inserted a new block                      number=153779171 hash=f93834…32c376 txs=64 gas=6089296 elapsed=518.712ms processTxs=115.996ms finalize=34.560ms validateState=1.534ms   totalWrite=18.641786ms trieWrite=18.265752ms
INFO[05/17,16:09:00 Z] [5] Inserted a new block                      number=153779172 hash=dd03f2…730106 txs=63 gas=3982520 elapsed=455.473ms processTxs=91.263ms  finalize=40.447ms validateState=1.473ms   totalWrite=96.461031ms trieWrite=95.886221ms
INFO[05/17,16:09:01 Z] [5] Inserted a new block                      number=153779173 hash=4968db…f36459 txs=9  gas=1017535 elapsed=392.102ms processTxs=31.903ms  finalize=16.352ms validateState=621.605µs totalWrite=18.34071ms  trieWrite=18.000548ms
INFO[05/17,16:09:01 Z] [5] Inserted a new block                      number=153779174 hash=a89094…fcbd7c txs=10 gas=1231369 elapsed=257.210ms processTxs=15.929ms  finalize=10.508ms validateState=372.61µs  totalWrite=8.983052ms  trieWrite=8.734766ms

Expected behavior
It should have full archival data.

Attachments
If applicable, add data, logs or screenshots to help explain your problem.

Environment (please complete the following information)

• Klaytn version or git revision that exhibits the issue $ ken version
• go version $ go version
• OS and its version [e.g. macOS/10.14]

root@2e420995eeb7:/var/kend/data# ken attach klay.ipc
Welcome to the Klaytn JavaScript console!

 instance: Klaytn/v1.12.1/linux-amd64/go1.20.6
  datadir: /var/kend/data
  modules: admin:1.0 debug:1.0 eth:1.0 governance:1.0 istanbul:1.0 klay:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0

Additional context
Add any other context about the problem here.

[ian0371]

Hi, did you download the chaindada from here?
It is not an archive data, it's a full data.

To run an archive node, you can either (1) start sync from block 0, or (2) use archive AMI on AWS.