Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect validator set hash #831

Closed
tomtau opened this issue Mar 22, 2021 · 10 comments · Fixed by #834
Closed

Incorrect validator set hash #831

tomtau opened this issue Mar 22, 2021 · 10 comments · Fixed by #834
Assignees
Labels
bug Something isn't working light-client Issues/features which involve the light client

Comments

@tomtau
Copy link
Contributor

tomtau commented Mar 22, 2021

I was trying Hermes CLI (v0.18.1) against one public testnet (running Tendermint v0.34.3) and got:

invalid light block: invalid validator set: header_validators_hash=CF04E331EDB7D559EBA9FE236FCA994A7C4C96BBAE1E840F6B3AFC0E0D065102 validators_hash=D657E7E4BF19C1DBCE99314EA94906A9DFE15CE101C8D3F9FC0F9774611CA417"}

Steps to reproduce

Feed some Croeseid public testnet block to the light client:

e.g. https://testnet-croeseid.crypto.org:26657/block?height=1190060

{
  "jsonrpc": "2.0",
  "id": -1,
  "result": {
    "block_id": {
      "hash": "B2325E299F1B7D8027D99EF4FAE2C0EBCD2ED38B64CDD280A5A2A18322481B8B",
      "parts": {
        "total": 1,
        "hash": "BC9F7B5A5BE8E9F296F9EE86057C404156190242A1CB5A79C4185009A503F346"
      }
    },
    "block": {
      "header": {
        "version": {
          "block": "11"
        },
        "chain_id": "testnet-croeseid-2",
        "height": "1190060",
        "time": "2021-03-22T07:07:54.376983365Z",
        "last_block_id": {
          "hash": "BA156F155AD3126CD9D36E47DEE724A6CF892B8E72FA0EE758850CB6BA6E7426",
          "parts": {
            "total": 1,
            "hash": "13F5973722B507AAD8A8A9834DB97B8835E66C1ECDFA8E4289E5C4664A8EBC55"
          }
        },
        "last_commit_hash": "CB323C7878111AD19A9E765686A02CEAFD7AEB9EA9D092309DA0267421BF1C5B",
        "data_hash": "6303FDB1365F3875E418F3DE88990802D6219301B70EA837DA995F7F79D92AB5",
        "validators_hash": "CF04E331EDB7D559EBA9FE236FCA994A7C4C96BBAE1E840F6B3AFC0E0D065102",
        "next_validators_hash": "CF04E331EDB7D559EBA9FE236FCA994A7C4C96BBAE1E840F6B3AFC0E0D065102",
        "consensus_hash": "048091BC7DDC283F77BFBF91D73C44DA58C3DF8A9CBC867405D8B7F3DAADA22F",
        "app_hash": "45CEC1606C82B6D9C29DDB0AA17A72A1911FDCECF85A3DA9BC8853C4479CEA7D",
        "last_results_hash": "04C4812ED690D60442BB1CF3551A8D2507D618EF189FEC98DEDE2EDD63E8E478",
        "evidence_hash": "E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855",
        "proposer_address": "15CDB1DCB71FE74CB7EA971E2CC17751F21984E8"
      },
      "data": {
        "txs": [
          "CpQBCpEBChwvY29zbW9zLmJhbmsudjFiZXRhMS5Nc2dTZW5kEnEKK3Rjcm8xeXIwcDN0M3V6OWZ4anpnbGUzbTY5dTUyZ2ttcW1nNm5wM2ZmZ3ISK3Rjcm8xeXIwcDN0M3V6OWZ4anpnbGUzbTY5dTUyZ2ttcW1nNm5wM2ZmZ3IaFQoIYmFzZXRjcm8SCTEwMDAwMDAwMBJuClMKRgofL2Nvc21vcy5jcnlwdG8uc2VjcDI1NmsxLlB1YktleRIjCiECBtaLa8I+u2ecdh8NuOL0GQSRNR4nj2UVZtjDrsKZTiUSBAoCCAEYpInIAhIXChEKCGJhc2V0Y3JvEgUyMDAwMBDAmgwaQBWSqxwmIIJDQEd7QCnGptGuNIZr3X37iIEgeMPXSuITB1DGkWwlvqURTuqaSRYlQop3Woi/JMOKiWY1TMHdqDw=",
          "CpQBCpEBChwvY29zbW9zLmJhbmsudjFiZXRhMS5Nc2dTZW5kEnEKK3Rjcm8xeXIwcDN0M3V6OWZ4anpnbGUzbTY5dTUyZ2ttcW1nNm5wM2ZmZ3ISK3Rjcm8xeXIwcDN0M3V6OWZ4anpnbGUzbTY5dTUyZ2ttcW1nNm5wM2ZmZ3IaFQoIYmFzZXRjcm8SCTEwMDAwMDAwMBJuClMKRgofL2Nvc21vcy5jcnlwdG8uc2VjcDI1NmsxLlB1YktleRIjCiECBtaLa8I+u2ecdh8NuOL0GQSRNR4nj2UVZtjDrsKZTiUSBAoCCAEYpYnIAhIXChEKCGJhc2V0Y3JvEgUyMDAwMBDAmgwaQJFiXiwyj+zQhFPftFKzcVER+LtgApuLmvry7snScPpXTJlKas3FiTB80ZOcjNAO0U7BMukDVNOEZew+LETVxpo=",
          "CpQBCpEBChwvY29zbW9zLmJhbmsudjFiZXRhMS5Nc2dTZW5kEnEKK3Rjcm8xeXIwcDN0M3V6OWZ4anpnbGUzbTY5dTUyZ2ttcW1nNm5wM2ZmZ3ISK3Rjcm8xeXIwcDN0M3V6OWZ4anpnbGUzbTY5dTUyZ2ttcW1nNm5wM2ZmZ3IaFQoIYmFzZXRjcm8SCTEwMDAwMDAwMBJuClMKRgofL2Nvc21vcy5jcnlwdG8uc2VjcDI1NmsxLlB1YktleRIjCiECBtaLa8I+u2ecdh8NuOL0GQSRNR4nj2UVZtjDrsKZTiUSBAoCCAEYponIAhIXChEKCGJhc2V0Y3JvEgUyMDAwMBDAmgwaQMCSY4BLz4dwZdQYgCVoLnqGFTIQldNq83CsnMJgmK69U3/BEAJdXf5ERaY1Rh+jDxRegSULXWwFea500OcGh28=",
          "Cp8BCpwBCiMvY29zbW9zLnN0YWtpbmcudjFiZXRhMS5Nc2dEZWxlZ2F0ZRJ1Cit0Y3JvMWRhOWdlOTNxN2ducjN2a2huMDZ3dDNqMHRycDI5ZnNjZHlwMmM1Ei90Y3JvY25jbDFkYTlnZTkzcTdnbnIzdmtobjA2d3QzajB0cnAyOWZzY2Ntem5xaBoVCghiYXNldGNybxIJNjgyOTUyODAyEm8KUgpGCh8vY29zbW9zLmNyeXB0by5zZWNwMjU2azEuUHViS2V5EiMKIQJTp//qSAbfRfBal7cCUEUpnW1YAZGIJs/TZ6DE6brDjxIECgIIARi4wwQSGQoSCghiYXNldGNybxIGODAwMDAwEICk6AMaQIZ/SXewgRgId66Tw2DSZzJGpS1+qkJkkEEkYjDxbzGHN0Qw6eKyD/SewQ+0JYXvWkJI1+flOHFxJlCY7u8vp2A=",
          "CpQBCpEBChwvY29zbW9zLmJhbmsudjFiZXRhMS5Nc2dTZW5kEnEKK3Rjcm8xeXIwcDN0M3V6OWZ4anpnbGUzbTY5dTUyZ2ttcW1nNm5wM2ZmZ3ISK3Rjcm8xeXIwcDN0M3V6OWZ4anpnbGUzbTY5dTUyZ2ttcW1nNm5wM2ZmZ3IaFQoIYmFzZXRjcm8SCTEwMDAwMDAwMBJuClMKRgofL2Nvc21vcy5jcnlwdG8uc2VjcDI1NmsxLlB1YktleRIjCiECBtaLa8I+u2ecdh8NuOL0GQSRNR4nj2UVZtjDrsKZTiUSBAoCCAEYp4nIAhIXChEKCGJhc2V0Y3JvEgUyMDAwMBDAmgwaQNs3h7gcC5w8axhB4yIYoTDFprjsqUcEodw/tietwtTPNdLghV0Sn5X1YS1VEyh2osrUGJ3YE9VFycYtf+usM44=",
          "CpQBCpEBChwvY29zbW9zLmJhbmsudjFiZXRhMS5Nc2dTZW5kEnEKK3Rjcm8xeXIwcDN0M3V6OWZ4anpnbGUzbTY5dTUyZ2ttcW1nNm5wM2ZmZ3ISK3Rjcm8xeXIwcDN0M3V6OWZ4anpnbGUzbTY5dTUyZ2ttcW1nNm5wM2ZmZ3IaFQoIYmFzZXRjcm8SCTEwMDAwMDAwMBJuClMKRgofL2Nvc21vcy5jcnlwdG8uc2VjcDI1NmsxLlB1YktleRIjCiECBtaLa8I+u2ecdh8NuOL0GQSRNR4nj2UVZtjDrsKZTiUSBAoCCAEYqInIAhIXChEKCGJhc2V0Y3JvEgUyMDAwMBDAmgwaQM/q2VLk7RuZphIGgGrRE4yoWh9T/CvsIiN5sVRsUfOeOBxvGMA4RyZOg5eGz2VQAdozNAJeOt1ToVGwP5E/3YM=",
          "CpQBCpEBChwvY29zbW9zLmJhbmsudjFiZXRhMS5Nc2dTZW5kEnEKK3Rjcm8xeXIwcDN0M3V6OWZ4anpnbGUzbTY5dTUyZ2ttcW1nNm5wM2ZmZ3ISK3Rjcm8xeXIwcDN0M3V6OWZ4anpnbGUzbTY5dTUyZ2ttcW1nNm5wM2ZmZ3IaFQoIYmFzZXRjcm8SCTEwMDAwMDAwMBJuClMKRgofL2Nvc21vcy5jcnlwdG8uc2VjcDI1NmsxLlB1YktleRIjCiECBtaLa8I+u2ecdh8NuOL0GQSRNR4nj2UVZtjDrsKZTiUSBAoCCAEYqYnIAhIXChEKCGJhc2V0Y3JvEgUyMDAwMBDAmgwaQDyzPynguWkFowCFkU7Zc+58t4213ev8smePqwZ5kjbeSY98yb/rrOETFpNJPzk0jGAvkTVKF1xoRhhj40O68c8=",
          "CpQBCpEBChwvY29zbW9zLmJhbmsudjFiZXRhMS5Nc2dTZW5kEnEKK3Rjcm8xeXIwcDN0M3V6OWZ4anpnbGUzbTY5dTUyZ2ttcW1nNm5wM2ZmZ3ISK3Rjcm8xeXIwcDN0M3V6OWZ4anpnbGUzbTY5dTUyZ2ttcW1nNm5wM2ZmZ3IaFQoIYmFzZXRjcm8SCTEwMDAwMDAwMBJuClMKRgofL2Nvc21vcy5jcnlwdG8uc2VjcDI1NmsxLlB1YktleRIjCiECBtaLa8I+u2ecdh8NuOL0GQSRNR4nj2UVZtjDrsKZTiUSBAoCCAEYqonIAhIXChEKCGJhc2V0Y3JvEgUyMDAwMBDAmgwaQMd08xVdTmrMm4iBBh0kkd0bbfW7HSh+xj4wwzXZsjeTC11dpvW7eDoTXrPyYxXOpmEPj/hF8wc/QgcGQ+i880E=",
          "CpQBCpEBChwvY29zbW9zLmJhbmsudjFiZXRhMS5Nc2dTZW5kEnEKK3Rjcm8xeXIwcDN0M3V6OWZ4anpnbGUzbTY5dTUyZ2ttcW1nNm5wM2ZmZ3ISK3Rjcm8xeXIwcDN0M3V6OWZ4anpnbGUzbTY5dTUyZ2ttcW1nNm5wM2ZmZ3IaFQoIYmFzZXRjcm8SCTEwMDAwMDAwMBJuClMKRgofL2Nvc21vcy5jcnlwdG8uc2VjcDI1NmsxLlB1YktleRIjCiECBtaLa8I+u2ecdh8NuOL0GQSRNR4nj2UVZtjDrsKZTiUSBAoCCAEYq4nIAhIXChEKCGJhc2V0Y3JvEgUyMDAwMBDAmgwaQC4Rsrzh5SfHJ6Aw7xVWQcIcvE591DWfWFVj54aL+Wj1JW7VL1mbFBPiHrhl+lyVDOF0PPl+u4L/4r6VWzY4WiU=",
          "CpQBCpEBChwvY29zbW9zLmJhbmsudjFiZXRhMS5Nc2dTZW5kEnEKK3Rjcm8xeXIwcDN0M3V6OWZ4anpnbGUzbTY5dTUyZ2ttcW1nNm5wM2ZmZ3ISK3Rjcm8xeXIwcDN0M3V6OWZ4anpnbGUzbTY5dTUyZ2ttcW1nNm5wM2ZmZ3IaFQoIYmFzZXRjcm8SCTEwMDAwMDAwMBJuClMKRgofL2Nvc21vcy5jcnlwdG8uc2VjcDI1NmsxLlB1YktleRIjCiECBtaLa8I+u2ecdh8NuOL0GQSRNR4nj2UVZtjDrsKZTiUSBAoCCAEYrInIAhIXChEKCGJhc2V0Y3JvEgUyMDAwMBDAmgwaQHL3aj47WOgX0t+dv9oIsKrJgS+IwVkiiiCejbGvCouUfvT3pAYZEnfTclpWYd2tQ47DiAhXt9VM/RFhOwEaplM=",
          "Cp8BCpwBCiMvY29zbW9zLnN0YWtpbmcudjFiZXRhMS5Nc2dEZWxlZ2F0ZRJ1Cit0Y3JvMW16NXJkdGY5d3Vmd2toOHRlMnp3dzd0d3RtbmE2cmhsMnFsaGxjEi90Y3JvY25jbDFtejVyZHRmOXd1ZndraDh0ZTJ6d3c3dHd0bW5hNnJobGxsdXc4bRoVCghiYXNldGNybxIJNDQ4Nzg4Mjc4EmwKUgpGCh8vY29zbW9zLmNyeXB0by5zZWNwMjU2azEuUHViS2V5EiMKIQKG+CNrIz6N99S0nClnTyMqLRhChvWRkRQ/8T1ASMwwbRIECgIIARikphASFgoQCghiYXNldGNybxIENTAwMBDAmgwaQPIXSSZLkSMz9aDPgu8yUoNZd4m4h+30gF7LlVx/NlhxRcVynoLbOA1amWyZJivaOVvbV0LwZh83cuGZuiWgDWE=",
          "CpQBCpEBChwvY29zbW9zLmJhbmsudjFiZXRhMS5Nc2dTZW5kEnEKK3Rjcm8xeXIwcDN0M3V6OWZ4anpnbGUzbTY5dTUyZ2ttcW1nNm5wM2ZmZ3ISK3Rjcm8xeXIwcDN0M3V6OWZ4anpnbGUzbTY5dTUyZ2ttcW1nNm5wM2ZmZ3IaFQoIYmFzZXRjcm8SCTEwMDAwMDAwMBJuClMKRgofL2Nvc21vcy5jcnlwdG8uc2VjcDI1NmsxLlB1YktleRIjCiECBtaLa8I+u2ecdh8NuOL0GQSRNR4nj2UVZtjDrsKZTiUSBAoCCAEYrYnIAhIXChEKCGJhc2V0Y3JvEgUyMDAwMBDAmgwaQP3HL3wqTHHAAqHtQUio2HvtIlQxpSeCO/U6cU3KXGNVRDlu3v+2t1WQQX0cvwMZ7haRQCbkNpopbAZDnY6FLC4=",
          "CpQBCpEBChwvY29zbW9zLmJhbmsudjFiZXRhMS5Nc2dTZW5kEnEKK3Rjcm8xeXIwcDN0M3V6OWZ4anpnbGUzbTY5dTUyZ2ttcW1nNm5wM2ZmZ3ISK3Rjcm8xeXIwcDN0M3V6OWZ4anpnbGUzbTY5dTUyZ2ttcW1nNm5wM2ZmZ3IaFQoIYmFzZXRjcm8SCTEwMDAwMDAwMBJuClMKRgofL2Nvc21vcy5jcnlwdG8uc2VjcDI1NmsxLlB1YktleRIjCiECBtaLa8I+u2ecdh8NuOL0GQSRNR4nj2UVZtjDrsKZTiUSBAoCCAEYronIAhIXChEKCGJhc2V0Y3JvEgUyMDAwMBDAmgwaQL+Iuulfg+bRshajSvGHAl3z/52fvHgU41J23t2pxcTJMn6DtejuKtFGpaxjDEqYDry8TFh40VWW9/EQy/Ldgik="
        ]
      },
      "evidence": {
        "evidence": []
      },
      "last_commit": {
        "height": "1190059",
        "round": 0,
        "block_id": {
          "hash": "BA156F155AD3126CD9D36E47DEE724A6CF892B8E72FA0EE758850CB6BA6E7426",
          "parts": {
            "total": 1,
            "hash": "13F5973722B507AAD8A8A9834DB97B8835E66C1ECDFA8E4289E5C4664A8EBC55"
          }
        },
        "signatures": [
          {
            "block_id_flag": 2,
            "validator_address": "703B26AEA0867B03572719D22F4B8E6D93CA838C",
            "timestamp": "2021-03-22T07:07:54.380658335Z",
            "signature": "iTiJBfxyJJgakz+QFbh7lMIUnVn9gMEOe2n7W/n4QaCoEkntmYzrJ1bQgmo8RLUBBswOzZ2O0u30NzfcU946Aw=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "0FB7AE9AC2E3F148CA130341B6CD4DB3682E2D54",
            "timestamp": "2021-03-22T07:07:54.490716448Z",
            "signature": "UCVE2fKWl6X4uFHnhBnJaqvajLg95+QcYor0WItgJ2jJCfW06wX7pyJ6Jyqy+ywXueXsmTifKPegJKpZA/B2AQ=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "AEA0F558C9616A7089791D1AE4C08DC5F69A0A0B",
            "timestamp": "2021-03-22T07:07:54.376983365Z",
            "signature": "8742a/eElroa2LxI2dGP7an68hgR1Ivwe6+CiSgb88PhTZ++DDpclOL7AMg36m72ygZmkdxviuRLDC/h1gxWCQ=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "15CDB1DCB71FE74CB7EA971E2CC17751F21984E8",
            "timestamp": "2021-03-22T07:07:54.120661218Z",
            "signature": "v/ZXz868LP2+h7dGpycXaFuGDmXgAFrqzRtdoPuv7Q+sc7Z2tiTfRB5eP1EZZno7BWS7BmU6EuI1RgXTg40MBg=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "0F190152E7788B88018569A951D1AAC383100A7E",
            "timestamp": "2021-03-22T07:07:54.184995714Z",
            "signature": "TKDXy3AN1vJlCO4G5ocbPA+SfAR+yJjCydBfu8Wc3RX8pc6CkBFis/SAAaDR8q3djezqbj01KACL1uh1XpwwAQ=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "1168CFC74D9EBE420C52349D67F967A38F01D69E",
            "timestamp": "2021-03-22T07:07:54.134140809Z",
            "signature": "/58GfQu5KbGcCaoS0qV5XqHTwl7abgV49avEu7tqNsDa5X1xTte7OXF7Yxf1Ymd7GNgr/5NqD35R9igq9Y0iCA=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "1CBA71D7ACCCCC529A2107D3891EFD5DD9537904",
            "timestamp": "2021-03-22T07:07:54.232688515Z",
            "signature": "ngJlBd4Nbz+0QHfC6048vfaTc07HsN1qJolnRikXd0t9ABogAyX3dLxpzN027GBzSk588CCeAIz/X/WtyVEfBQ=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "73F3DAAE50A59F67990065ECAA33BF89F49C44B0",
            "timestamp": "2021-03-22T07:07:54.282463538Z",
            "signature": "esHNp+mSTrbjSDpZurBqUuSLbTVN/4zvWvojeY3bDNOtS55pFPfq9Ei19viyRU2j3OCVxULgu1QzsLoQPbMpBA=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "A2477C39D01A74C97D42C13509B70F50D867D09E",
            "timestamp": "2021-03-22T07:07:54.16003746Z",
            "signature": "rv9KjCbHXjGdPHy02mrs6HqHgo9H+qFzyFaKo493tj22E4rrTlARbaEexF7JHEWbLnYJCoQVSZhRjiWynprhAA=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "47D58509899DCEF91835CD24AAB8DBE4BAB5F8C3",
            "timestamp": "2021-03-22T07:07:54.177205Z",
            "signature": "vUSEUmTeptG0X0EiVhOwVuGwx80c1Cm7odKlvCNTj/IltKRG+nXFKCL93omJl6SjF/aStD7998nfOsgxnMNpDA=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "61D387F1302D7E5CAD042D7C399455911FBA63A5",
            "timestamp": "2021-03-22T07:07:54.177869555Z",
            "signature": "L9lTL1mKDWDDKZ3UEUY7XjnXBefoVNE1jzRkz48jYcUIdnM+e254AUimEDWq4/XZsW2ptLmEJcY4991ldQ9CCg=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "B8D8AAD0DF9FBEBA4A2A29E21A421C8E86FADD67",
            "timestamp": "2021-03-22T07:07:54.132247375Z",
            "signature": "G48aFyfGURw8lGxRZQ3KR69js1lfsP7QMxKOjW82vIzugEvDMw+v7SzGSKqZjsGNkrWgy5sNHt9LwdsZibXKDw=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "98C15BB3D81C57DA801E0E62AB1CB0E1E7C7EEB9",
            "timestamp": "2021-03-22T07:07:54.17294466Z",
            "signature": "jTR/dLc/kntAqx59fWK7Wwh82L5FKwIPn/n8H4+SArnw7oySmkyXSoBSUfXT9Mc7X9wAMGiiRiJ/K+4KkCHWDw=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "CF237EDC88CBFF3FE95B8077FDF1F1E6CBA5BB1B",
            "timestamp": "2021-03-22T07:07:54.140206064Z",
            "signature": "w20ZuyINKwhXJnzIxyx5c7iqBYX1S3CkHJCWH1lQHtiFBzqTr3fpTunkKlcpsGZo7CzKzOoRQa77MnMKe4A/AA=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "63CC9E48A80F2C192AECDFC5D5C7B9775FB4B4AB",
            "timestamp": "2021-03-22T07:07:54.139108629Z",
            "signature": "+aZWm32Gakiuj97J000WIULxE9wQwA5IA/Raygg/lAwD19a3m4vSZwtOBu4ZroE16iLv58RnaJMdztv2fbVoAg=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "DE6BAC083B63F2C6DE8A696597CA78CB0677C902",
            "timestamp": "2021-03-22T07:07:54.140699618Z",
            "signature": "rrasmetbCwv0cxoGDxTpXNZhy9KAp2lF90nCjCPvmrrSxu1lJcFaQ7Pgb1UyUtkNYfX0bp0Wz0Mny5X0drnmAg=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "A3DC2532609F7C2E6BBAB2809080F9EA1EA183C0",
            "timestamp": "2021-03-22T07:07:54.185003018Z",
            "signature": "CCxim2hO3q3m4/kNHhVndm/XseuemPYvQuzdBRflfsk2YeA7R+70hS1nN/69hOBTyPUrMdXXRqWW5OLGAhbeDg=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "95CDD1C2F0E79F62745D17A90D9A7B138DC8F922",
            "timestamp": "2021-03-22T07:07:54.191317352Z",
            "signature": "j2m4QPHw+q5huEKt9/eaRrvCmg7hF3wcb3fREhsmNV7mLmTJM8iDxl0t8zRawP0aKAjkUxo62j6OdPJSm+d1CA=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "D95491EF01B523570556CEE5A60A14AAB02C828C",
            "timestamp": "2021-03-22T07:07:54.274860577Z",
            "signature": "hnnTAwesMvRJWRhjV5vMM0fbyAsp4Vx7Jq5UEfMHtt09r4lfZISqCO5vATmkBETs4awOG+qx4Jldoh3P0ZzECA=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "50B54C1E37BB9383558FC5FD04BE69E3B229FE20",
            "timestamp": "2021-03-22T07:07:54.138893689Z",
            "signature": "FYJwEmdJX/MEDpA8a62I7n4jw4J3lKZXgNl2I0zCCbmXlGIaNW0Vn7Tr36OYxGtiFoYaLLjkXl9FHYCYC6d/Cg=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "B9E7788C01EDAEEC04A6A819701B932117B13060",
            "timestamp": "2021-03-22T07:07:54.239288732Z",
            "signature": "peM7s6A+Wutqa00O0hwwC2KRmI7YYSaniqCoRycpgexjpHsagRQv3QilcgnF5GDovQtPJMAqT3BMFOdETQYBDA=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "7929D3222D14C63E9785BD19EDEBBCDC2244D8AA",
            "timestamp": "2021-03-22T07:07:54.176485105Z",
            "signature": "CL1xlEpK0nx6Jmk7l47YECbh2uyW8ziKlse3UiADzRae5ZfQ1EvwKZyMpQStL+PUPtAE5u2QxAos06wdlJ/WBw=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "A1E8AAEBBC82929B852748734BA39D67A62F201B",
            "timestamp": "2021-03-22T07:07:54.274112164Z",
            "signature": "dPrquDL77IYCJJ6SPsypoIB7Kmy98f0KxZ3ZtdvYZigCeQ1DtsAvdoPDVib+P2z6BAKBHa8dsf4gAEytBjPkAw=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "50AA09728D618C1C632D1E6C8BF0F399062CBA26",
            "timestamp": "2021-03-22T07:07:54.150186083Z",
            "signature": "5UrbhjRT8V17DSAhNkRYt7qN6fbphLCGM7qfqsHCe2qxYZ5R4WACaJ5KCHnr79GNLdWY3be4wwbk2ruNnRr4Ag=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "5B4A2A6A239AD792F48A0CDD5469D563E0E2B786",
            "timestamp": "2021-03-22T07:07:54.187587434Z",
            "signature": "h7a/OYmISBf1J57EhpGKPoICWTu33yJMWHymIdoqoglqzISr8mhPqtKwLZQFvWkm4XzJvTpLK4TEdcJ7SZfMBg=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "EDB80454CA0459949E6E564A979E2A5C20BE2DAC",
            "timestamp": "2021-03-22T07:07:54.275172801Z",
            "signature": "eCCaHXzVuqaiHTU24btQz6NNMeZP7I8sLaXst/06n3cy9MquFkVXK4gaw6IXa6Cb5EdHUdEUr4penwZpDmqOBg=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "E23F5C63108A87067478C426B7CE3B8DABFC1ED4",
            "timestamp": "2021-03-22T07:07:54.244437303Z",
            "signature": "mL6dErt2U7RP9x6G6zMF5DhBuUXLTTzWfKJ+PwS0l2fAGJz3Uk/RoobBQsj6WxWRtwcw0Kk3eeR71PFTK8LlBQ=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "6E66EC70E6A243F69BC82AEF3783E63E237A6FE1",
            "timestamp": "2021-03-22T07:07:54.26359935Z",
            "signature": "MRiLm0HHtI0o5yhGUpLh7jUZI9lUh5inzhJrxeLFoYGLoomW4A9tc+rloFn25A21SzWQfUjnsu7gWa1l4Eb1BA=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "0F7479FBE0193349D3FD22D4F5ADD2BFFFAB1DA1",
            "timestamp": "2021-03-22T07:07:54.205714264Z",
            "signature": "b1QVi4TEmrEBs7z9JGWbOmt6e70G5dZinLoK1ASnJLWXhODFdOZVpTnrvEAU/gRbWf5pSUIbKwQadnacx2WFCA=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "725BFF0129AE55F287366DE568940FD76280F180",
            "timestamp": "2021-03-22T07:07:54.190246294Z",
            "signature": "FAAoKQt8w8k7iBxHzggpYLth+g1E1Iv1ZfYynSwnR6uRbj71rrH4LLZAWpwlOxPcDp8EVkMJhXfKSnRi9nJ5AA=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "D3A418F7C83C7DE225D6F5C9E8E97CC054FD1469",
            "timestamp": "2021-03-22T07:07:54.244369714Z",
            "signature": "DD4QClo/dScQI06mKy8D4bUGa0IMUxa39xWSxGXJP7WI9ZOxlwgSr/a85DmCbN38iORf1MWg8GVUACauxDtKBw=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "62E0A1625735852A49D0D42BE3D11EE6EE4EB198",
            "timestamp": "2021-03-22T07:07:54.191627674Z",
            "signature": "0xQpW1Sgzkr0Enun7EAzUUqehAAKr/yy5qcbd/KnnbiQW0cTzo8/yRNjlJ5DSSQDmgE5vmWtL5yUrsHj2uQyDg=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "C0775C24181955E41DCA6C2B573B26DB90675C85",
            "timestamp": "2021-03-22T07:07:54.486590974Z",
            "signature": "LDrWfCrmrtJtG412rcjzoo3imQQDZDOpEPVYRQ1CB7vIE+TIqOZbtKcmtN8Hh89OHFFdr1FAYx/9TPkNV/qkDQ=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "E7517C3DE6A3B83DA856E15D5F1892A22FB4CE23",
            "timestamp": "2021-03-22T07:07:54.225945499Z",
            "signature": "IEQYKs1aZfIsep52nWMha9jsQzpvfdkj0vhRL3e1yLB2Xd73bmsD0U0WPJ56PvWMh9rjpRDNiUWtxEKTu/CyAw=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "BD89D5F926C89A09AB5ADF655C909197ED1778E1",
            "timestamp": "2021-03-22T07:07:54.177411875Z",
            "signature": "Ficc5M+zdc3ssyFtL0aZ0OgrsIITpjp/Rm7XBTerHF5HtyuMZKHFEFqGdtfHO1Hcb0rOc6SQ+vWiCorC6WBuDg=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "F6E63F456C8EA56BF8908D7D3FDA6B2D4CCB1D85",
            "timestamp": "2021-03-22T07:07:54.194558598Z",
            "signature": "EnfWbbTvd+2WI10FLGQg77++/nTC/NTNThysH/U6NnOwrWNeXU3DEk0NJZfJw8M8rRni/itpP2uMsTeMVF0+Cg=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "205834921063B2A7C92DC6DA863AC81693E47493",
            "timestamp": "2021-03-22T07:07:54.279430262Z",
            "signature": "3IkQI1v5Anee3ZCpTjFDC+5LtTJe47uduTKFRQ6BtFZgaTO4qPfPM0xkowz4amsw1zZYYI4vrj1zUsfMkRJtCA=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "14B0AC9BF54275D17BE5B42B75EB0E68785B952D",
            "timestamp": "2021-03-22T07:07:54.1917345Z",
            "signature": "dH37v9+CecbGNeZxRh0NYqHQgs67vS4mYFe6g25kawBESpxu16KuXn0pqjQreLquFTf+XPUbiHVrO50sYcPgBQ=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "AB4600EE62C2C7A55BB00BFD2288CD4F196FB35D",
            "timestamp": "2021-03-22T07:07:54.194879055Z",
            "signature": "krJujbH+LKSuINt3nKRzlXz93WfL+KgFKhdztm1jStBM9z4DzRdv0K2QjEglAOidUYRH4gzdWz2SC6wHVOVBCg=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "6E145AFA7F451447074CBC17D3E892232614F5D2",
            "timestamp": "2021-03-22T07:07:54.243094776Z",
            "signature": "ZTG1xv3tjHzEaDePKfcVJmEHYQHlpSraan+r5C6QjcwwBzAH7AEvkedFRYCvmuFs2Yyuet9TIh2UzEYi04PsDQ=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "3FFD072EA3413023A50125272ED7069F3DB1F290",
            "timestamp": "2021-03-22T07:07:54.180671779Z",
            "signature": "WDs3hZzBv4W+kcm9st+RIVUihTgnrkHlxqt8xIT4W3jst1o6PAV5aXWCv8cTmUjEDNiqzAaQvrIFcl1zu6UiDg=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "4E06C4F0C2B2D21C4D08361B4590D6CA531A4035",
            "timestamp": "2021-03-22T07:07:54.281337607Z",
            "signature": "j9Bi91ggphyDU+RlxvqLwE9tsrXxXVBrvTA3R+LpnYRN3FP/+ZBpZObwbyyNestev/C8GuH+0oUyyAYaacQJCQ=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "5B56AC9E2B5A130200F0DB094B90EB1512A7C80E",
            "timestamp": "2021-03-22T07:07:54.200939319Z",
            "signature": "a1kodj1fNofLbvtjvin2BnoETDsXZ7ZlNB245BPdMjblOpGc4AuC7B8QPdUOWMxqlishWTA1MGopJFenTaoLBA=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "54056CB5F2FFB3F2681B8DD2647DF59AA1CBA033",
            "timestamp": "2021-03-22T07:07:54.232002743Z",
            "signature": "TBQO8Vgo3/1GouIQg288anchJS13zetne80zQU9k2JMP+IGabqULrH53VKm919wKbj9yxlo4HRfJCks0HyblAg=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "5D7B2D44CDAC899167C44AD3E3286D6DA80A31DE",
            "timestamp": "2021-03-22T07:07:54.183196803Z",
            "signature": "t4vJNJrEWdW5v/CLNK+JZ2Ydyxt9vsC1FOkfB+qBI/MB0XdHWwiK8f2+GHIwhIws5KgVGSTf2PkTUQ6u0LP6AQ=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "EB2BFCF783236F6E1793CF8568CBBFA46D5C146C",
            "timestamp": "2021-03-22T07:07:54.134172068Z",
            "signature": "Tprbh2juHuXyGLv81FYD18+iWeJbgp44X35e8pTGQmB4+LH8Cy2P+DPSNb45hx56JwUsNtWxpKzW9K2m4d0+Cw=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "203AA5FB460704E308B49F43208A912C645B2096",
            "timestamp": "2021-03-22T07:07:54.189641634Z",
            "signature": "FR/y980cDMKafiIaYX0p3tgkNZ596eGlY+XpHXOsrGpmH0pWtwMS1P+rd4CUZTDoq07k1eN7xW/WwxCyuOB9CQ=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "48D281AB1ECB445DD357DEDF69E7771A4F719730",
            "timestamp": "2021-03-22T07:07:54.214985216Z",
            "signature": "GrYBTZVQc77Hjh9c/Lk/e718ZlM+HYjDVPwnuywI2P0xo8XCuNTscqwHkWFt2HkZHVgHEdhSAwjjSRgSgoz7AA=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "F5B94FF43496A36DE19BE60ACC9D3C3BF6A40C1A",
            "timestamp": "2021-03-22T07:07:54.225197537Z",
            "signature": "WCZEYv0KPviE3nKHuyHEaQZAga7ig1lDefglhhaVXt/eUI0v+DODWfsv6udBRCMTyxV1872Seke5uTN4GEGcCQ=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "51E350AACE36B1EAEA6859CE667A82DB74BBCE03",
            "timestamp": "2021-03-22T07:07:54.207173638Z",
            "signature": "iZejCS3aFa++9rn77rBZx138Ic9iBUE/uxPzvg0M8fATtNHTLIyW3+/KRebYmd97qC2zObCwKcH/gF7BxNIzDA=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "671DAC1E6CDADBD830D9454C88D7183BD6851B39",
            "timestamp": "2021-03-22T07:07:54.213147188Z",
            "signature": "qKZXCXtZM9pEihs2ywr+X3RzGogLMVZ38KzGDNvEWtQhay34qWrFI9y/bkHAPrclnsCiTbpFPzvacRzAqMzPBg=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "5F2297AFFAB722F2702BC0E710E5A2E26712F147",
            "timestamp": "2021-03-22T07:07:54.190155307Z",
            "signature": "a0Mfif470gt7D4ASRNaI9zm4lk7guzSnNCznjQVbwxkCwWdUS3XQ810weATuSwlR3yHqOP9KExV4A9NekZJTCg=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "B9006823320C5D1297B44390BC7FBFA3EE59C185",
            "timestamp": "2021-03-22T07:07:59.763757257Z",
            "signature": "ySeLBcXhdXPwbnjIbaoQCE4HvVNQZWaA0YbP6in2ejGjiiHcNx1F9vanhTdkUgmG9tLhmGz1krXniD+Lcw+RAg=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "AC1CB5FDF0DB39FF9329C4D3C54153D6C201828E",
            "timestamp": "2021-03-22T07:07:54.209018965Z",
            "signature": "JekUeGKa4eP26LBVM9KdJ+44MCFrjprUB9WR9wi/QcF7nuuGCY6+69rVvDiPGAevc3PCWiQVSxxVsY/A15nrBg=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "12333286BB4161FDA97887EF99B364A20F88D525",
            "timestamp": "2021-03-22T07:07:54.302833437Z",
            "signature": "oXM9iUFZmfq6UretvIH7QSo9VOepUJ70BT0cGPhibLRGWxAavqT4CCv4Abu1AvtdGycOe2aOuhT3sDA5/cMUBw=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "2C6695AB2F86216E81E531AB3514A2E320E76972",
            "timestamp": "2021-03-22T07:07:54.11844136Z",
            "signature": "iUwRxpVm+MLtloiQFF3vnGYgKquVKevDi3D63s/Z1g6N0iRwMH6m4BwkDCE8+kPtm56oK7vhD2Hw2vf/N2TUAA=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "E8F2755E1824B53C5B63F637A439F7901CDDAAF4",
            "timestamp": "2021-03-22T07:07:54.201614475Z",
            "signature": "doLxquzoAEitgNv+2pXMWoM+GCkrj/fbga6ikKby5PNm9Nb6mUQtJgy6JalZVZF3Gn2HWlnQ0P716WFi4mejBA=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "CC8A7F5B7AFA2ACA8CD52EFC4549650B418BF604",
            "timestamp": "2021-03-22T07:07:54.12945594Z",
            "signature": "cq2N+983xRvUiYrh5g9gRp47T2rIjlaX70Ui8kyaYElXZ9Rj9/l8F00SPm3tveZCIjbYMVZvYNlrdzRq87BsCg=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "ADB2B2FB5CA60ADA7E29D9B55B59CB491F492E8F",
            "timestamp": "2021-03-22T07:07:54.137464914Z",
            "signature": "2XHLpHKKYAbGOSWUY44Vs5IE2jNj0eSVpT0gXgDYcvI/jrOvbHRU6N3FRAzJgu0aE7J1iS0LVF/AYDEzaNLsDg=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "D7E11A8E3A43D6D3B305CDA52239B4F1275BD2AF",
            "timestamp": "2021-03-22T07:07:54.399399096Z",
            "signature": "/k32cF/I1lGPIQUgF9+Lr+HqvrqpwXOl6mV8pHGQZ6p5XP7R1T+o4tY22ZwnFgfKAY4NbbCUekmpqcXWSgq8BA=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "9CE42B371C847CCED82777F65976D375F52F43C4",
            "timestamp": "2021-03-22T07:07:54.327654245Z",
            "signature": "YKzVIENO019NPwLxP1yAcAINpeWEunnX6YAAy22aHj1PelUCQb6OhqguKWqyQqYKxu4FJJVY1xgTiYWek3g4BA=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "B3456BF7EE0EA368E4B9A318E5FDBA78C76454C3",
            "timestamp": "2021-03-22T07:07:54.330635087Z",
            "signature": "ZiLFT70ot5CaS+HKKzoI441+UbvxwiMy/33SFjMNjyD9hnha/EnlRY+PsqoMD6qTOgiTWqqbquyKZGLlzp1pBQ=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "102119E620DA8CBFFB55ACF1A784248BFBC1DA9A",
            "timestamp": "2021-03-22T07:07:54.2808593Z",
            "signature": "JsleoztLGwRxHEnsb1IClD6E9CZ2tSLvtfRsEa2zwG7AfMQXk6S3v2W8kzWuV980vyfrXbmI5mrWodC2ox+DCg=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "5FDD7333CC652E7090F63AEC2EB37885AB2D7D2E",
            "timestamp": "2021-03-22T07:07:54.17997843Z",
            "signature": "ZZxMVHZcWQgzpgX5RrrApgrTA109wA+Q9L4WRwU+a45LrGb/c5Z5h5huus+7/kx75V54TkedrL2NvcU8ToR/AA=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "8B44E0F47DDD37538CC823A308A440747809B900",
            "timestamp": "2021-03-22T07:07:54.21492827Z",
            "signature": "61sjf+A2uR0W/jwwi1Fmj1/I0xkplkouRY38Oa0Ji3T2zUEaH/Gh5gLd11RXi09SMEis+WyBHG4Njy/CU9WNAQ=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "1E2DDEB76E3165F107932A2BDF893BFEE7CEDA71",
            "timestamp": "2021-03-22T07:07:54.25701787Z",
            "signature": "W9XQokLLcdaDWHMaU5Z9K8J+EBuzm6/aZjhpCUyVw8MFkD4TVuhM+94tdRCrLYVDGwS53eZWjF8MwT4+ZDafAA=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "1B0BDF62E33953C1F796C40DF79FF342D84D7960",
            "timestamp": "2021-03-22T07:07:54.242178207Z",
            "signature": "ikGtqio0hsWXA5R44wVK30I8jI5Pqno6YPz/R6qs4j9rqiBdqlu/EeMv6uHHnTDY4KkOTkKb5jK/ImyI2LfyDA=="
          },
          {
            "block_id_flag": 2,
            "validator_address": "8A0FD69195284A09AD53970A190E8C5100F95AA1",
            "timestamp": "2021-03-22T07:07:54.34403625Z",
            "signature": "O9Z3v06/D0iP/UvNomNHsMENZ85lnmVL3RKrgkkgiD7IWVMCbKFIVaq7aBVZkDdk1lO+rNKJA2JqRQ2riTvnCw=="
          }
        ]
      }
    }
  }
}

and the corresponding validators: https://testnet-croeseid.crypto.org:26657/validators?height=1190060

{
  "jsonrpc": "2.0",
  "id": -1,
  "result": {
    "block_height": "1190060",
    "validators": [
      {
        "address": "703B26AEA0867B03572719D22F4B8E6D93CA838C",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "Og8ZfQTHFgTBGD5qoyo5NpyJCJRddC+WuSPtyZtlE7E="
        },
        "voting_power": "3950084749",
        "proposer_priority": "6592168648"
      },
      {
        "address": "0FB7AE9AC2E3F148CA130341B6CD4DB3682E2D54",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "wWw0e9tZcVmev/NyJlZv5Apd7U5IONoyx3U/9rD5fHI="
        },
        "voting_power": "3571314752",
        "proposer_priority": "-4358797356"
      },
      {
        "address": "AEA0F558C9616A7089791D1AE4C08DC5F69A0A0B",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "BuuPYme7R4eH/nWs2p+sS1UpCQwy+QJgBZuhGICH8Es="
        },
        "voting_power": "3459400600",
        "proposer_priority": "9032208757"
      },
      {
        "address": "15CDB1DCB71FE74CB7EA971E2CC17751F21984E8",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "4vImlt6TzxU+OZZ5882bHh6vB6JX1dUiDB/fo5w4Sk8="
        },
        "voting_power": "1515794704",
        "proposer_priority": "-7842534143"
      },
      {
        "address": "0F190152E7788B88018569A951D1AAC383100A7E",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "FdAGNCx/v5wkn8pRv8E+1vsCcS6BpUsg1w7Wd1Vbz50="
        },
        "voting_power": "826966270",
        "proposer_priority": "-3404998762"
      },
      {
        "address": "1168CFC74D9EBE420C52349D67F967A38F01D69E",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "w1TP1Zp8YlYLo1QFn3olnuRiHeCwBcAhGzHZT6uf11E="
        },
        "voting_power": "779942356",
        "proposer_priority": "-7501463120"
      },
      {
        "address": "1CBA71D7ACCCCC529A2107D3891EFD5DD9537904",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "mzcsEkZrb93HvWcl9Yv2b6CWIgN+7MxTCR2Xjmhi2Xc="
        },
        "voting_power": "748416330",
        "proposer_priority": "-7123079589"
      },
      {
        "address": "73F3DAAE50A59F67990065ECAA33BF89F49C44B0",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "YSrv0PcW6pOb2IYZv2+23iGaWFt8SDutFVKAPAPoRk4="
        },
        "voting_power": "737060564",
        "proposer_priority": "4984605686"
      },
      {
        "address": "A2477C39D01A74C97D42C13509B70F50D867D09E",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "Osswoe/7pqnvBLl3gEY1rMGc0sZtNNI5vpLe9U7SEts="
        },
        "voting_power": "490022547",
        "proposer_priority": "-2572936198"
      },
      {
        "address": "47D58509899DCEF91835CD24AAB8DBE4BAB5F8C3",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "cl6+4AKFu8rvmTVTJ77ue36pMMdMoiX0LcczkWwxnqw="
        },
        "voting_power": "386500500",
        "proposer_priority": "9027888994"
      },
      {
        "address": "61D387F1302D7E5CAD042D7C399455911FBA63A5",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "wGe8AEihKSvqQ5pN7aGACVPR3nqPmuhct7DitQiZC0w="
        },
        "voting_power": "378657000",
        "proposer_priority": "7043017911"
      },
      {
        "address": "B8D8AAD0DF9FBEBA4A2A29E21A421C8E86FADD67",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "AkVapYCLhXbK//h24TydEXcIFkry3d2WCxVWhmggfqk="
        },
        "voting_power": "292800299",
        "proposer_priority": "699105337"
      },
      {
        "address": "98C15BB3D81C57DA801E0E62AB1CB0E1E7C7EEB9",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "91Ttn3cnr4slVMtVB9abzaVB+/ORjsWE8WoL2S/vtFE="
        },
        "voting_power": "289712252",
        "proposer_priority": "-5273707478"
      },
      {
        "address": "CF237EDC88CBFF3FE95B8077FDF1F1E6CBA5BB1B",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "u3Nv2LTEcg/smBjtT11rSBD7I+Ddhf8AJIjEMQUkl5o="
        },
        "voting_power": "202451878",
        "proposer_priority": "792998559"
      },
      {
        "address": "63CC9E48A80F2C192AECDFC5D5C7B9775FB4B4AB",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "Pw9ZUJzUayfTfolnfatKCiuIfH/P+FKUMqz48RYg1l0="
        },
        "voting_power": "167723610",
        "proposer_priority": "2230879474"
      },
      {
        "address": "DE6BAC083B63F2C6DE8A696597CA78CB0677C902",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "nIRwu1Y8Gxegz/9WGje6AWtd8LGdgzdbGnpzXDXE1ds="
        },
        "voting_power": "164427643",
        "proposer_priority": "13177506"
      },
      {
        "address": "A3DC2532609F7C2E6BBAB2809080F9EA1EA183C0",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "Ie2EZ2VbnxhvWHB9G/zbCASIxDhHZlo4VOPvmEvLmhg="
        },
        "voting_power": "155906257",
        "proposer_priority": "1820347690"
      },
      {
        "address": "95CDD1C2F0E79F62745D17A90D9A7B138DC8F922",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "k4Rhz4ASR7lteJoZfsNUTOmm0+8giSYbLGsrEvmHzSA="
        },
        "voting_power": "155619243",
        "proposer_priority": "-4278184709"
      },
      {
        "address": "D95491EF01B523570556CEE5A60A14AAB02C828C",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "Uaabtkw5cSDmRFVozQbya9VRsHoSSo304hFANkzluUg="
        },
        "voting_power": "151633283",
        "proposer_priority": "-7039525173"
      },
      {
        "address": "50B54C1E37BB9383558FC5FD04BE69E3B229FE20",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "THCXja8CqmZUJoLxXj1lzNZ8EANtCKRW1YGMjNHk5ng="
        },
        "voting_power": "141947713",
        "proposer_priority": "-7359670441"
      },
      {
        "address": "B9E7788C01EDAEEC04A6A819701B932117B13060",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "844OSYPpgslCz/gMzZM5uG+vgrJh2/CpEZUbv4sPpjE="
        },
        "voting_power": "107703167",
        "proposer_priority": "7827129705"
      },
      {
        "address": "7929D3222D14C63E9785BD19EDEBBCDC2244D8AA",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "L7QZEzMZlz6GfQf6CNAqaqnu7CxTaXzu+hSsRH0qPiA="
        },
        "voting_power": "104031090",
        "proposer_priority": "-492938557"
      },
      {
        "address": "A1E8AAEBBC82929B852748734BA39D67A62F201B",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "i0NL59+ZEKyqSrqvXBwD7UhVQ3y68kxxTZerEe7APAg="
        },
        "voting_power": "101029056",
        "proposer_priority": "3594515756"
      },
      {
        "address": "50AA09728D618C1C632D1E6C8BF0F399062CBA26",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "weyEZz7OG/JNae8iCb5aGO1Yo9mFccK4UgWhALYrULY="
        },
        "voting_power": "87267456",
        "proposer_priority": "-8060890892"
      },
      {
        "address": "5B4A2A6A239AD792F48A0CDD5469D563E0E2B786",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "0I3SAHq/DlwSQTCGi8odDd08Xjn90gjY4G1BnA7IDfk="
        },
        "voting_power": "84403432",
        "proposer_priority": "3465254337"
      },
      {
        "address": "EDB80454CA0459949E6E564A979E2A5C20BE2DAC",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "xjCMA8ogVt8Gp0JoNalK0d65iJljNEHnd+aG7N0Jr64="
        },
        "voting_power": "81007252",
        "proposer_priority": "-3171051990"
      },
      {
        "address": "E23F5C63108A87067478C426B7CE3B8DABFC1ED4",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "YF5RlzKhq0KUreG18W3NiOMpyvXexmQko3PYFqsdcwQ="
        },
        "voting_power": "80885281",
        "proposer_priority": "5106380077"
      },
      {
        "address": "6E66EC70E6A243F69BC82AEF3783E63E237A6FE1",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "npqpOwpsJt1toWsBa+Vj/pYXwQXuMuIKvFafgoHqhLk="
        },
        "voting_power": "80767493",
        "proposer_priority": "2786830063"
      },
      {
        "address": "0F7479FBE0193349D3FD22D4F5ADD2BFFFAB1DA1",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "8lvrVtgZPIkviDtd+2r0gLsNvFCWWdgIeKeM9xpNiWI="
        },
        "voting_power": "80704412",
        "proposer_priority": "-6199594040"
      },
      {
        "address": "725BFF0129AE55F287366DE568940FD76280F180",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "Scg3ZOX7i8wKE8aXyoQRehY8EU/Y1gW5XCzmJgKfDuI="
        },
        "voting_power": "80680450",
        "proposer_priority": "1327097326"
      }
    ],
    "count": "30",
    "total": "68"
  }
}

What's the definition of "done" for this issue?

The validator set hash is calculated correctly -- not sure if this is a duplicate of this issue: #687

@tomtau tomtau added the bug Something isn't working label Mar 22, 2021
@tomtau
Copy link
Contributor Author

tomtau commented Mar 22, 2021

one other suspect here is "paging":

    "count": "30",
    "total": "68"

@ancazamfir
Copy link
Contributor

ancazamfir commented Mar 23, 2021

Running into this when light client validates a trusted block during initially instance building for some hermes operations (client create or client update). The two chains are cosmoshub-4 and akashnet-2.
The relevant trace is:

   6: tendermint_light_client::builder::light_client::LightClientBuilder<tendermint_light_client::builder::light_client::NoTrustedState>::validate::{{closure}}
             at /Users/ancaz/.cargo/registry/src/github.com-1ecc6299db9ec823/tendermint-light-client-0.18.1/src/builder/light_client.rs:186:26
   7: core::result::Result<T,E>::map_err
             at /Users/ancaz/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/core/src/result.rs:595:27
   8: tendermint_light_client::builder::light_client::LightClientBuilder<tendermint_light_client::builder::light_client::NoTrustedState>::validate
             at /Users/ancaz/.cargo/registry/src/github.com-1ecc6299db9ec823/tendermint-light-client-0.18.1/src/builder/light_client.rs:184:9
   9: tendermint_light_client::builder::light_client::LightClientBuilder<tendermint_light_client::builder::light_client::NoTrustedState>::trust_light_block
             at /Users/ancaz/.cargo/registry/src/github.com-1ecc6299db9ec823/tendermint-light-client-0.18.1/src/builder/light_client.rs:123:9
  10: tendermint_light_client::builder::light_client::LightClientBuilder<tendermint_light_client::builder::light_client::NoTrustedState>::trust_primary_at
             at /Users/ancaz/.cargo/registry/src/github.com-1ecc6299db9ec823/tendermint-light-client-0.18.1/src/builder/light_client.rs:169:9
  11: ibc_relayer::light_client::tendermint::build_instance
             at relayer/src/light_client/tendermint.rs:104:9

Looks like something goes wrong here

fn validator_sets_match(
&self,
light_block: &LightBlock,
hasher: &dyn Hasher,
) -> Result<(), VerificationError> {
let validators_hash = hasher.hash_validator_set(&light_block.validators);

--->

fn hash_validator_set(&self, validator_set: &ValidatorSet) -> Hash {
let validator_bytes: Vec<Vec<u8>> = validator_set
.validators()
.iter()
.map(|validator| validator.hash_bytes())
.collect();
Hash::Sha256(merkle::simple_hash_from_byte_vectors(validator_bytes))
}
}

[ nb: temp unrawp() at relayer/src/light_client/tendermint.rs:104]
full trace at https://gist.github.com/ancazamfir/66006b0c0feee151a1577834cb32081b

@ancazamfir
Copy link
Contributor

Could we feed a unit test with a real chain block and debug this please?

@thanethomson thanethomson added the light-client Issues/features which involve the light client label Mar 23, 2021
@thanethomson
Copy link
Member

thanethomson commented Mar 24, 2021

As @tomtau pointed out, it looks like this happens when there are more than 30 validators in the validator set, due to the following:

  1. The validators Tendermint RPC endpoint is paged (https://docs.tendermint.com/master/rpc/#/Info/validators). It returns 30 validators by default, if the request doesn't include the desired number of validators. See https://github.com/tendermint/tendermint/blob/d43a79ddf6d13eecdcba5735b5d086900dcea988/rpc/core/types/responses.go#L139
  2. The tendermint-rs RPC client doesn't provide options to request a specific page at present. See
    pub struct Request {
  3. Responses will naturally be truncated at 30 validators, resulting in the hash being calculated from an incomplete set of validators.

This should be fixed if we allow for paging in the validators RPC request, and implement paging in the Light Client where the request originates (over here).

@thanethomson thanethomson self-assigned this Mar 24, 2021
@Shivani912
Copy link
Contributor

The validator set hash is calculated correctly -- not sure if this is a duplicate of this issue: #687

#687 originates from the MBT tool so I don't suspect this to be the issue here @tomtau

@romac
Copy link
Member

romac commented Mar 24, 2021

@thanethomson Could we get an option on rpc::HttpClient::validators to specify paging?

Something like this perhaps:

async fn validators<H>(&self, height: H, paging: Paging) -> Result<validators::Response> { /* ... */ }

enum Paging {
  NoPaging,
  AllPages,
  Paged {
    page: usize,
    per_page: usize,
  }
}

impl Default for Paging {
  fn default() -> Self {
    Self::NoPaging
  }
}

alternatively:

async fn validators<H>(&self, height: H, paging: Option<Paging>) -> Result<validators::Response> { /* ... */ }

enum Paging {
  AllPages,
  Paged {
    page: usize,
    per_page: usize,
  }
}

@jackzampolin
Copy link

Fwiw I'm also seeing something that looks like this in the go relayer

@thanethomson
Copy link
Member

Fwiw I'm also seeing something that looks like this in the go relayer

Yeah we spoke about this earlier today. The pagination's definitely an issue on the tendermint-rs side, but it might not be the only issue - we'll know once I fix the pagination issue 🙂

@colin-axner
Copy link

The pagination's definitely an issue on the tendermint-rs side, but it might not be the only issue

I believe this is the case. The go relayer was incorrectly pulling the trusted validator set from the trusted height and it looks like hermes might have been doing the same. The on-chain consensus state stores the hash for the NextValidatorSet, so we should grab the header for trusted_height+1 and use that validator set

Ironically the ibc-go tests happened to be doing this correctly

It makes sense why this was a hard issue to track down. It was only a problem when the validator set changed between trusted height X and X + 1

@ancazamfir
Copy link
Contributor

I believe this is the case. The go relayer was incorrectly pulling the trusted validator set from the trusted height and it looks like hermes might have been doing the same. The on-chain consensus state stores the hash for the NextValidatorSet, so we should grab the header for trusted_height+1 and use that validator set

Ironically the ibc-go tests happened to be doing this correctly

It makes sense why this was a hard issue to track down. It was only a problem when the validator set changed between trusted height X and X + 1

thanks Colin!!! opened informalsystems/hermes#770 to fix this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working light-client Issues/features which involve the light client
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants