Indexer Erroring out with Duplicate Entries for Production Build

[sufiyanghori]

Hi,

We are trying to run data processing pipeline in Production mode.

Everything seems to be running alright, however, when we check Production Logs, we continue to see following,

indexer_1      | [2021-02-11T21:23:52.46329294Z]: job:bootstrap event:dbr.exec time:1746 μs kvs:[sql:INSERT INTO `address_chain` (`address`,`chain_id`,`created_at`) VALUES ('JYecDLqzKt7DUtrmLXnuk55k2o561AjW7','11111111111111111111111111111111LpoYY','2020-09-10 00:00:00.000000')]
indexer_1      | [2021-02-11T21:23:52.465773084Z]: job:bootstrap event:dbr.exec.exec err:Error 1062: Duplicate entry 'VD4KE3To81AfgGR4RGNooLhR49vRXt3UbWmzskgDuUn27isyK-JYecDLqzKt7DUt' for key 'avm_output_addresses.avm_output_addresses_output_id_addr' kvs:[sql:INSERT INTO `avm_output_addresses` (`output_id`,`address`,`created_at`) VALUES ('VD4KE3To81AfgGR4RGNooLhR49vRXt3UbWmzskgDuUn27isyK','JYecDLqzKt7DUtrmLXnuk55k2o561AjW7','2020-09-10 00:00:00.000000')]
indexer_1      | [2021-02-11T21:23:52.465882991Z]: job:bootstrap event:dbr.exec time:2 ms kvs:[sql:INSERT INTO `avm_output_addresses` (`output_id`,`address`,`created_at`) VALUES ('VD4KE3To81AfgGR4RGNooLhR49vRXt3UbWmzskgDuUn27isyK','JYecDLqzKt7DUtrmLXnuk55k2o561AjW7','2020-09-10 00:00:00.000000')]
indexer_1      | [2021-02-11T21:23:52.466110647Z]: job:bootstrap event:dbr.exec time:3 ms kvs:[sql:INSERT INTO `avm_output_addresses` (`output_id`,`address`,`created_at`) VALUES ('VD4KE3To81AfgGR4RGNooLhR49vRXt3UbWmzskgDuUn27isyK','JYecDLqzKt7DUtrmLXnuk55k2o561AjW7','2020-09-10 00:00:00.000000')]
indexer_1      | [2021-02-11T21:23:52.469401347Z]: job:bootstrap event:dbr.exec.exec err:Error 1062: Duplicate entry '2urSTLvfPC1GpRBvprQvgt4VTB2HJUrxA7iTSW7aFYWSxobN1A' for key 'avm_outputs.PRIMARY' kvs:[sql:INSERT INTO `avm_outputs` (`id`,`chain_id`,`transaction_id`,`output_index`,`asset_id`,`output_type`,`amount`,`locktime`,`threshold`,`group_id`,`payload`,`stake_locktime`,`stake`,`frozen`,`created_at`) VALUES ('2urSTLvfPC1GpRBvprQvgt4VTB2HJUrxA7iTSW7aFYWSxobN1A','11111111111111111111111111111111LpoYY','2k7yszSgGEw25wbK7DApfg9A181NysrTjN245YviqRCWZHcujs',126,'FvwEAhmxKfeiG8SnEvq42hc6whRyY3EFYAvebMqDNDGCgxN5Z',7,1642588235294,0,1,0,?,1638576000,1,0,'2020-09-10 00:00:00.000000')]
indexer_1      | [2021-02-11T21:23:52.469488256Z]: job:bootstrap event:dbr.exec time:3 ms kvs:[sql:INSERT INTO `avm_outputs` (`id`,`chain_id`,`transaction_id`,`output_index`,`asset_id`,`output_type`,`amount`,`locktime`,`threshold`,`group_id`,`payload`,`stake_locktime`,`stake`,`frozen`,`created_at`) VALUES ('2urSTLvfPC1GpRBvprQvgt4VTB2HJUrxA7iTSW7aFYWSxobN1A','11111111111111111111111111111111LpoYY','2k7yszSgGEw25wbK7DApfg9A181NysrTjN245YviqRCWZHcujs',126,'FvwEAhmxKfeiG8SnEvq42hc6whRyY3EFYAvebMqDNDGCgxN5Z',7,1642588235294,0,1,0,?,1638576000,1,0,'2020-09-10 00:00:00.000000')]
indexer_1      | [2021-02-11T21:23:52.469850122Z]: job:bootstrap event:dbr.exec time:3 ms kvs:[sql:INSERT INTO `avm_outputs` (`id`,`chain_id`,`transaction_id`,`output_index`,`asset_id`,`output_type`,`amount`,`locktime`,`threshold`,`group_id`,`payload`,`stake_locktime`,`stake`,`frozen`,`created_at`) VALUES ('2urSTLvfPC1GpRBvprQvgt4VTB2HJUrxA7iTSW7aFYWSxobN1A','11111111111111111111111111111111LpoYY','2k7yszSgGEw25wbK7DApfg9A181NysrTjN245YviqRCWZHcujs',126,'FvwEAhmxKfeiG8SnEvq42hc6whRyY3EFYAvebMqDNDGCgxN5Z',7,1642588235294,0,1,0,?,1638576000,1,0,'2020-09-10 00:00:00.000000')]

The containers are freshly started with MySQL having no data.

Can someone please point us what we are doing wrong?

[tasinco]

The message: Error 1062: Duplicate entry -- is not an actual error.

[sufiyanghori]

The message: Error 1062: Duplicate entry -- is not an actual error.

Thank you @tasinco for your response. But when we query the indexer around that time, we always get 0,

curl "http://localhost:8080/v2/aggregates?startTime=2020-09-21T00:00:00Z&endTime=2020-10-21T00:00:00Z"

{
  "aggregates": {
    "startTime": "2020-09-21T00:00:00Z",
    "endTime": "2020-10-21T00:00:00Z",
    "transactionVolume": "0",
    "transactionCount": 0,
    "addressCount": 0,
    "outputCount": 0,
    "assetCount": 0
  },
  "startTime": "2020-09-21T00:00:00Z",
  "endTime": "2020-10-21T00:00:00Z"
}

[sufiyanghori]

The message: Error 1062: Duplicate entry -- is not an actual error.

There is no data after 2020-09-10 00:00:00.000000 and thats the date in the error too. Can you please suggest whats the issue?

[tasinco]

start time is inclusive end time is not..
remove endtime or increase it into the future..

[sufiyanghori]

start time is inclusive end time is not..
remove endtime or increase it into the future..

Even if we try without the endTime, we don't get any results after 2020-09-10T00:00:00Z

Please have a look,

[root@ip-172-31-27-19 ~]# curl "http://localhost:8080/v2/aggregates?startTime=2020-09-10T00:00:00Z"
{"aggregates":{"startTime":"2020-09-10T00:00:00Z","endTime":"2021-02-11T22:08:40Z","transactionVolume":"359999999999990210","transactionCount":77,"addressCount":4936,"outputCount":38844,"assetCount":1},"startTime":"2020-09-10T00:00:00Z","endTime":"2021-02-11T22:08:40Z"}

[root@ip-172-31-27-19 ~]#
[root@ip-172-31-27-19 ~]#
[root@ip-172-31-27-19 ~]#
[root@ip-172-31-27-19 ~]# curl "http://localhost:8080/v2/aggregates?startTime=2020-09-11T00:00:00Z"

{"aggregates":{"startTime":"2020-09-11T00:00:00Z","endTime":"2021-02-11T22:08:52Z","transactionVolume":"0","transactionCount":0,"addressCount":0,"outputCount":0,"assetCount":0},"startTime":"2020-09-11T00:00:00Z","endTime":"2021-02-11T22:08:52Z"}
[root@ip-172-31-27-19 ~]#

[tasinco]

The duplicate row was not inserted.. The first row that did insert set the timestamp. You would need the timestamp of the first row. This is still not an issue.

[sufiyanghori]

The duplicate row was not inserted.. The first row that did insert set the timestamp. You would need the timestamp of the first row. This is still not an issue.

Thank you @tasinco for your response. I can totally understand what you are saying that duplicate rows were not inserted. I am confused to understand that why the subsequent rows are not inserted either? The ones after the duplicated row, and how can I resume inserting the rows?

[tasinco]

Subsequent non duplicated rows will be inserted.. It is possible that you have not had any new rows added possibly because the node is still bootstrapping.
You can query the db to check what rows were added.

[Craddock7]

Let the team at GitHub fix your issues asap

[tasinco]

You can review: https://github.com/ava-labs/ortelius/blob/master/docs/production.md <- on how to setup ortelius.