Unable to setup Observer

[Chapoly1305]

Goal
Clone the chain to local so I can review history information for research purpose. Specifically, I am trying to observe the trend of device models of Matter, and the OtaURL of each models on chain.

Attempted
Followed,

1. https://github.com/zigbee-alliance/distributed-compliance-ledger/blob/master/docs/running-node-manual/prerequisites.md
2. https://github.com/zigbee-alliance/distributed-compliance-ledger/blob/master/docs/running-node-manual/full-node.md
3. https://github.com/zigbee-alliance/distributed-compliance-ledger/blob/master/docs/running-node-manual/on.md

Environment
Ubuntu 22.04 w/ public IP address, firewall allowed ingress.

Configuration Used

• genesis.json for main-net
• app.toml.txt
• client.toml.txt
• config.toml.txt

Acceptable Alternative
If not possible to resolve, appreciate helps for providing a guide to check from any existed server.

Occurrence
Unable to sync any blocks to local, logs complained "error: wrong Block.Header.AppHash"

curl http://localhost:26657/status | jq .result.sync_info
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1081  100  1081    0     0   474k      0 --:--:-- --:--:-- --:--:-- 1055k
{
  "latest_block_hash": "B9F3E76529081EECDCA011F89826C7D1FBD4CAB6E8F9485323B9EE57DE522846",
  "latest_app_hash": "E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855",
  "latest_block_height": "1",
  "latest_block_time": "2022-08-11T15:49:35.603086763Z",
  "earliest_block_hash": "B9F3E76529081EECDCA011F89826C7D1FBD4CAB6E8F9485323B9EE57DE522846",
  "earliest_app_hash": "E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855",
  "earliest_block_height": "1",
  "earliest_block_time": "2022-08-11T15:49:35.603086763Z",
  "catching_up": true
}

May 05 21:23:37 1pb.us cosmovisor[506566]: 9:23PM INF service start impl=MConn{124.220.76.45:29780} module=p2p msg="Starting MConnection service" peer=b77151736898951aed3f130eb35ea154dc8e8cef@124.220.76.45:29780
May 05 21:23:37 1pb.us cosmovisor[506566]: 9:23PM INF service start impl="Peer{MConn{35.80.181.38:9952} 3c6a644f374ee89172fcba7ea81a076e606a1c2d in}" module=p2p msg="Starting Peer service" peer=3c6a644f374ee89172fcba7ea81a076e606a1c2d@35.80.181.38:9952
May 05 21:23:37 1pb.us cosmovisor[506566]: 9:23PM INF service start impl=MConn{35.80.181.38:9952} module=p2p msg="Starting MConnection service" peer=3c6a644f374ee89172fcba7ea81a076e606a1c2d@35.80.181.38:9952
May 05 21:23:39 1pb.us cosmovisor[506566]: 9:23PM INF service start impl="Peer{MConn{63.35.3.70:53781} ed887679725b9e07a18f34b5607bd7a10a201fed in}" module=p2p msg="Starting Peer service" peer=ed887679725b9e07a18f34b5607bd7a10a201fed@63.35.3.70:53781
May 05 21:23:39 1pb.us cosmovisor[506566]: 9:23PM INF service start impl=MConn{63.35.3.70:53781} module=p2p msg="Starting MConnection service" peer=ed887679725b9e07a18f34b5607bd7a10a201fed@63.35.3.70:53781
May 05 21:23:39 1pb.us cosmovisor[506566]: 9:23PM INF service start impl="Peer{MConn{13.124.79.22:30704} 87d3446c3927599d6485fa63df501b7fd2341f5a in}" module=p2p msg="Starting Peer service" peer=87d3446c3927599d6485fa63df501b7fd2341f5a@13.124.79.22:30704
May 05 21:23:39 1pb.us cosmovisor[506566]: 9:23PM INF service start impl=MConn{13.124.79.22:30704} module=p2p msg="Starting MConnection service" peer=87d3446c3927599d6485fa63df501b7fd2341f5a@13.124.79.22:30704
May 05 21:23:39 1pb.us cosmovisor[506566]: 9:23PM INF service start impl="Peer{MConn{3.101.123.198:57668} 26cd64e9391cd690fe29e97ad9871531b73c1d73 in}" module=p2p msg="Starting Peer service" peer=26cd64e9391cd690fe29e97ad9871531b73c1d73@3.101.123.198:57668
May 05 21:23:39 1pb.us cosmovisor[506566]: 9:23PM INF service start impl=MConn{3.101.123.198:57668} module=p2p msg="Starting MConnection service" peer=26cd64e9391cd690fe29e97ad9871531b73c1d73@3.101.123.198:57668
May 05 21:23:41 1pb.us cosmovisor[506566]: 9:23PM INF service start impl="Peer{MConn{34.215.208.72:63187} 7b910c0fe159676fd730148eb9efa64f11d0b8dd in}" module=p2p msg="Starting Peer service" peer=7b910c0fe159676fd730148eb9efa64f11d0b8dd@34.215.208.72:63187
May 05 21:23:41 1pb.us cosmovisor[506566]: 9:23PM INF service start impl=MConn{34.215.208.72:63187} module=p2p msg="Starting MConnection service" peer=7b910c0fe159676fd730148eb9efa64f11d0b8dd@34.215.208.72:63187
May 05 21:23:42 1pb.us cosmovisor[506566]: 9:23PM INF service start impl="Peer{MConn{18.201.110.189:47374} 78fabf3886d22e54d1f37aa3952b2eb8638fb130 in}" module=p2p msg="Starting Peer service" peer=78fabf3886d22e54d1f37aa3952b2eb8638fb130@18.201.110.189:47374
May 05 21:23:42 1pb.us cosmovisor[506566]: 9:23PM INF service start impl=MConn{18.201.110.189:47374} module=p2p msg="Starting MConnection service" peer=78fabf3886d22e54d1f37aa3952b2eb8638fb130@18.201.110.189:47374
May 05 21:23:43 1pb.us cosmovisor[506566]: 9:23PM INF service start impl="Peer{MConn{52.33.2.192:22711} 1e941f34b402c2e9b5b7d2d13184a331cba301ff in}" module=p2p msg="Starting Peer service" peer=1e941f34b402c2e9b5b7d2d13184a331cba301ff@52.33.2.192:22711
May 05 21:23:43 1pb.us cosmovisor[506566]: 9:23PM INF service start impl=MConn{52.33.2.192:22711} module=p2p msg="Starting MConnection service" peer=1e941f34b402c2e9b5b7d2d13184a331cba301ff@52.33.2.192:22711
May 05 21:23:44 1pb.us cosmovisor[506566]: 9:23PM ERR Error in validation err="wrong Block.Header.AppHash.  Expected 70EFFA65BD626F99B0C38C69A851B4DADF8DC7AEB8F7DA85321714D506F18836, got 310F35E60DF10CD05E92D69C8A4624A9B8D808E0214C874FEB04C505A9407EF1" module=blockchain
May 05 21:23:44 1pb.us cosmovisor[506566]: 9:23PM ERR Stopping peer for error err="Reactor validation error: wrong Block.Header.AppHash.  Expected 70EFFA65BD626F99B0C38C69A851B4DADF8DC7AEB8F7DA85321714D506F18836, got 310F35E60DF10CD05E92D69C8A4624A9B8D808E0214C874FEB04C505A9407EF1" module=p2p peer="Peer{MConn{18.201.110.189:47374} 78fabf3886d22e54d1f37aa3952b2eb8638fb130 in}"
May 05 21:23:44 1pb.us cosmovisor[506566]: 9:23PM INF service stop impl=MConn{18.201.110.189:47374} module=p2p msg="Stopping MConnection service" peer=78fabf3886d22e54d1f37aa3952b2eb8638fb130@18.201.110.189:47374
May 05 21:23:44 1pb.us cosmovisor[506566]: 9:23PM INF service stop impl="Peer{MConn{18.201.110.189:47374} 78fabf3886d22e54d1f37aa3952b2eb8638fb130 in}" module=p2p msg="Stopping Peer service" peer=78fabf3886d22e54d1f37aa3952b2eb8638fb130@18.201.110.189:47374
May 05 21:23:44 1pb.us cosmovisor[506566]: 9:23PM ERR Stopping peer for error err="Reactor validation error: wrong Block.Header.AppHash.  Expected 70EFFA65BD626F99B0C38C69A851B4DADF8DC7AEB8F7DA85321714D506F18836, got 310F35E60DF10CD05E92D69C8A4624A9B8D808E0214C874FEB04C505A9407EF1" module=p2p peer="Peer{MConn{3.101.123.198:57668} 26cd64e9391cd690fe29e97ad9871531b73c1d73 in}"
May 05 21:23:44 1pb.us cosmovisor[506566]: 9:23PM INF service stop impl=MConn{3.101.123.198:57668} module=p2p msg="Stopping MConnection service" peer=26cd64e9391cd690fe29e97ad9871531b73c1d73@3.101.123.198:57668
May 05 21:23:44 1pb.us cosmovisor[506566]: 9:23PM ERR Stopping peer for error err="read tcp 104.168.35.238:26656->3.101.123.198:57668: use of closed network connection" module=p2p peer="Peer{MConn{3.101.123.198:57668} 26cd64e9391cd690fe29e97ad9871531b73c1d73 in}"
May 05 21:23:44 1pb.us cosmovisor[506566]: 9:23PM INF service stop impl="Peer{MConn{3.101.123.198:57668} 26cd64e9391cd690fe29e97ad9871531b73c1d73 in}" module=p2p msg="Stopping Peer service" peer=26cd64e9391cd690fe29e97ad9871531b73c1d73@3.101.123.198:57668
May 05 21:23:44 1pb.us cosmovisor[506566]: 9:23PM ERR error while stopping peer error="already stopped" module=p2p
May 05 21:23:47 1pb.us cosmovisor[506566]: 9:23PM INF service start impl="Peer{MConn{13.52.115.12:52918} add9e1972b68487af3d64c79ab9883746f1a95f8 in}" module=p2p msg="Starting Peer service" peer=add9e1972b68487af3d64c79ab9883746f1a95f8@13.52.115.12:52918
May 05 21:23:47 1pb.us cosmovisor[506566]: 9:23PM INF service start impl=MConn{13.52.115.12:52918} module=p2p msg="Starting MConnection service" peer=add9e1972b68487af3d64c79ab9883746f1a95f8@13.52.115.12:52918
May 05 21:23:47 1pb.us cosmovisor[506566]: 9:23PM INF service start impl="Peer{MConn{52.39.248.59:34459} e9b8390ec08ccc2e945543b399681cd78e1594dd in}" module=p2p msg="Starting Peer service" peer=e9b8390ec08ccc2e945543b399681cd78e1594dd@52.39.248.59:34459
May 05 21:23:47 1pb.us cosmovisor[506566]: 9:23PM INF service start impl=MConn{52.39.248.59:34459} module=p2p msg="Starting MConnection service" peer=e9b8390ec08ccc2e945543b399681cd78e1594dd@52.39.248.59:34459
May 05 21:23:48 1pb.us cosmovisor[506566]: 9:23PM INF service start impl="Peer{MConn{18.144.69.78:34924} e8450750b01cdeefd30d7ec4e00a8daa71b4283e in}" module=p2p msg="Starting Peer service" peer=e8450750b01cdeefd30d7ec4e00a8daa71b4283e@18.144.69.78:34924
May 05 21:23:48 1pb.us cosmovisor[506566]: 9:23PM INF service start impl=MConn{18.144.69.78:34924} module=p2p msg="Starting MConnection service" peer=e8450750b01cdeefd30d7ec4e00a8daa71b4283e@18.144.69.78:34924
May 05 21:23:48 1pb.us cosmovisor[506566]: 9:23PM INF service start impl="Peer{MConn{52.204.175.98:59070} 5f09c719eb3bde3c922c19314b0f90632d1f0e1d in}" module=p2p msg="Starting Peer service" peer=5f09c719eb3bde3c922c19314b0f90632d1f0e1d@52.204.175.98:59070
May 05 21:23:48 1pb.us cosmovisor[506566]: 9:23PM INF service start impl=MConn{52.204.175.98:59070} module=p2p msg="Starting MConnection service" peer=5f09c719eb3bde3c922c19314b0f90632d1f0e1d@52.204.175.98:59070
May 05 21:23:50 1pb.us cosmovisor[506566]: 9:23PM INF service start impl="Peer{MConn{54.184.188.109:45902} 91c2bda2317c79bd0f44987380f5ec8630bdb182 in}" module=p2p msg="Starting Peer service" peer=91c2bda2317c79bd0f44987380f5ec8630bdb182@54.184.188.109:45902
May 05 21:23:50 1pb.us cosmovisor[506566]: 9:23PM INF service start impl=MConn{54.184.188.109:45902} module=p2p msg="Starting MConnection service" peer=91c2bda2317c79bd0f44987380f5ec8630bdb182@54.184.188.109:45902
May 05 21:23:50 1pb.us cosmovisor[506566]: 9:23PM ERR Error in validation err="wrong Block.Header.AppHash.  Expected 70EFFA65BD626F99B0C38C69A851B4DADF8DC7AEB8F7DA85321714D506F18836, got 310F35E60DF10CD05E92D69C8A4624A9B8D808E0214C874FEB04C505A9407EF1" module=blockchain
May 05 21:23:50 1pb.us cosmovisor[506566]: 9:23PM ERR Stopping peer for error err="Reactor validation error: wrong Block.Header.AppHash.  Expected 70EFFA65BD626F99B0C38C69A851B4DADF8DC7AEB8F7DA85321714D506F18836, got 310F35E60DF10CD05E92D69C8A4624A9B8D808E0214C874FEB04C505A9407EF1" module=p2p peer="Peer{MConn{18.144.69.78:34924} e8450750b01cdeefd30d7ec4e00a8daa71b4283e in}"
May 05 21:23:50 1pb.us cosmovisor[506566]: 9:23PM INF service stop impl=MConn{18.144.69.78:34924} module=p2p msg="Stopping MConnection service" peer=e8450750b01cdeefd30d7ec4e00a8daa71b4283e@18.144.69.78:34924
May 05 21:23:50 1pb.us cosmovisor[506566]: 9:23PM INF service stop impl="Peer{MConn{18.144.69.78:34924} e8450750b01cdeefd30d7ec4e00a8daa71b4283e in}" module=p2p msg="Stopping Peer service" peer=e8450750b01cdeefd30d7ec4e00a8daa71b4283e@18.144.69.78:34924

Thank you for your help!

[Chapoly1305]

Full service log.

service.log

[jcps07]

@Chapoly1305 were you able to figure this out?

[Chapoly1305]

@jcps07 Yes. Turns out there are multiple protocol revision and the tool isn't backward compatible.
You must clone with lower version of tool first for init, and it will break after sync x blocks. Then you need to use another version it suggested, and it will break after y blocks. The process repeated serval times. I will later feed you more details.