Constraint error on byEventId during sync

[bwindels]

Error in status bar:

Sync failed because of Error: wrapped DOMException: (name: ConstraintError) (code: ConstraintError) Unable to add key to index 'byEventId': at least one key does not satisfy the uniqueness requirements.

In console:

Uncaught (in promise) Error: add on undefined failed: (name: ConstraintError) (code: ConstraintError) Unable to add key to index 'byEventId': at least one key does not satisfy the uniqueness requirements.
    at Store.add (brawl.js:3685)
8brawl.js:3685 Uncaught (in promise) Error: add on undefined failed: (name: AbortError) (code: AbortError) The transaction was aborted, so the request cannot be fulfilled.
    at Store.add (brawl.js:3685)
brawl.js:3676 Uncaught (in promise) Error: put on undefined failed: (name: AbortError) (code: AbortError) The transaction was aborted, so the request cannot be fulfilled.
    at Store.put (brawl.js:3676)
brawl.js:686 unable to commit sync tranaction
_syncRequest @ brawl.js:686

[bwindels]

Again, on FF:

starting sync request with since s1510473512_757284957_1265062_573655706_401495312_1508011_85252908_260298148_124438 ... Sync.js:94:25
 * applying sync response to room !AZozoWghOYSIAfaZjJ:matrix.org ... Sync.js:144:29
unable to commit sync tranaction Sync.js:162:21
newStatus SyncError SessionStatusViewModel.js:91:21
Uncaught (in promise) Error: add on undefined failed: (name: ConstraintError) (code: ConstraintError) A mutation operation in the transaction failed because a constraint was not satisfied.
    StorageError http://localhost:3000/src/matrix/storage/common.js:44
    add http://localhost:3000/src/matrix/storage/idb/Store.js:126
common.js:44:9
Uncaught (in promise) Error: put on undefined failed: (name: AbortError) (code: AbortError) A request was aborted, for example through a call to IDBTransaction.abort.
    StorageError http://localhost:3000/src/matrix/storage/common.js:44
    put http://localhost:3000/src/matrix/storage/idb/Store.js:117
common.js:44:9
opening the timeline for !WCHwNGGsrVwPeRtAnq:matrix.org

[bwindels]

Even when you manage to get the value that failed to add in FF, the idb tools are quite horrible (doesn't show index, can't search properly) so it's still hard to figure out why the index (or store?) throws up a ConstraintError, although my guess would be that the key is already there somehow as in the first error?

We already guard against:

• existing events in a /messages response
• duplicate events within a single sync response

Perhaps we'll also have to check if any of the event ids in the sync response are already present during sync. Although it would be nice to catch synapse in the act if this would be the case.

[bwindels]

Caught FF in the act again:

Uncaught (in promise) Error: add on brawl_session_7158167632539208.timelineEvents failed: (name: ConstraintError) (code: 0) (value: {
  "fragmentId": 0,
  "eventIndex": 2147483657,
  "roomId": "!YxKeAxtNcDZDrGgaMF:matrix.org",
  "event": {
    "content": {
      "body": "lol",
      "msgtype": "m.text"
    },
    "origin_server_ts": 1597841747142,
    "sender": "@bwindels:matrix.org",
    "type": "m.room.message",
    "unsigned": {
      "age": 7326177
    },
    "event_id": "$RSJbbD8AdLvbA-6LdXpcDQNHExQ6dD3yFuVFRLidDQ8"
  },
  "key": "!YxKeAxtNcDZDrGgaMF:matrix.org|00000000|80000009",
  "eventIdKey": "!YxKeAxtNcDZDrGgaMF:matrix.org|$RSJbbD8AdLvbA-6LdXpcDQNHExQ6dD3yFuVFRLidDQ8"
}) A mutation operation in the transaction failed because a constraint was not satisfied.

with stack trace:

StorageError http://localhost:3000/src/matrix/storage/common.js:50
    add http://localhost:3000/src/matrix/storage/idb/Store.js:126
    insert http://localhost:3000/src/matrix/storage/idb/stores/TimelineEventStore.js:241
    _writeTimeline http://localhost:3000/src/matrix/room/timeline/persistence/SyncWriter.js:150
    writeSync http://localhost:3000/src/matrix/room/timeline/persistence/SyncWriter.js:181
    writeSync http://localhost:3000/src/matrix/room/Room.js:47
    promises http://localhost:3000/src/matrix/Sync.js:145
    parseRooms http://localhost:3000/src/matrix/Sync.js:39
    parseRooms http://localhost:3000/src/matrix/Sync.js:38
    _syncRequest http://localhost:3000/src/matrix/Sync.js:134
    _syncLoop http://localhost:3000/src/matrix/Sync.js:95
    start http://localhost:3000/src/matrix/Sync.js:86
    _waitForFirstSync http://localhost:3000/src/matrix/SessionContainer.js:183
    _loadSessionInfo http://localhost:3000/src/matrix/SessionContainer.js:163
    startWithExistingSession http://localhost:3000/src/matrix/SessionContainer.js:76
    createAndStartSessionContainer http://localhost:3000/src/domain/SessionPickerViewModel.js:160
    start http://localhost:3000/src/domain/SessionLoadViewModel.js:40
    pick http://localhost:3000/src/domain/SessionPickerViewModel.js:174
    onItemClick http://localhost:3000/src/ui/web/login/SessionPickerView.js:97
    _onClick http://localhost:3000/src/ui/web/general/ListView.js:88
    mount http://localhost:3000/src/ui/web/general/ListView.js:67
    view http://localhost:3000/src/ui/web/general/TemplateView.js:282
    render http://localhost:3000/src/ui/web/login/SessionPickerView.js:109
    mount http://localhost:3000/src/ui/web/general/TemplateView.js:95
    mount http://localhost:3000/src/ui/web/general/SwitchView.js:25
    mount http://localhost:3000/src/ui/web/BrawlView.js:54
    main http://localhost:3000/src/main.js:71
    async* http://localhost:3000/:21

The event is not in the timeline yet though:

So my guess is that the eventIndex is not advanced (or we're using the wrong fragmentIndex). Note that this happens right after start, so it might be a bug in SyncWriter.load, which reported this for the given room:

room persister load !YxKeAxtNcDZDrGgaMF:matrix.org [0/2147483656] SyncWriter.js:56:17

[bwindels]

2147483656 or 0x80000008

[bwindels]

So

bar

is stored as 0x80000008, and a sync event

lol

comes and is assigned 0x80000009, but somehow that fails?

[bwindels]

Aha!
I just had a look in an export of the db, and bar is 0x80000008, the foo before it is 0x80000009:

{
        "fragmentId": 0,
        "eventIndex": 2147483656,
        "roomId": "!YxKeAxtNcDZDrGgaMF:matrix.org",
        "event": {
          "content": {
            "body": "foo",
            "msgtype": "m.text"
          },
          "origin_server_ts": 1597828699861,
          "sender": "@bruno4:matrix.org",
          "type": "m.room.message",
          "unsigned": {
            "age": 2546303
          },
          "event_id": "$gttsUPJ509OeMgqT3OkHqdRNpAKFhLgvGNuJf_2UMrY"
        },
        "key": "!YxKeAxtNcDZDrGgaMF:matrix.org|00000000|80000008",
        "eventIdKey": "!YxKeAxtNcDZDrGgaMF:matrix.org|$gttsUPJ509OeMgqT3OkHqdRNpAKFhLgvGNuJf_2UMrY"
      },
      {
        "fragmentId": 0,
        "eventIndex": 2147483657,
        "roomId": "!YxKeAxtNcDZDrGgaMF:matrix.org",
        "event": {
          "content": {
            "body": "bar",
            "msgtype": "m.text"
          },
          "origin_server_ts": 1597828702928,
          "sender": "@bruno4:matrix.org",
          "type": "m.room.message",
          "unsigned": {
            "age": 2543236
          },
          "event_id": "$mTY998SnZXdKMejq1pJtSZibGfOb6rmtMWOIVhMxt0o"
        },
        "key": "!YxKeAxtNcDZDrGgaMF:matrix.org|00000000|80000009",
        "eventIdKey": "!YxKeAxtNcDZDrGgaMF:matrix.org|$mTY998SnZXdKMejq1pJtSZibGfOb6rmtMWOIVhMxt0o"
      },

so likely load is not fetching the last key correctly?

[bwindels]

The last comments might be related to a problem introduced in the bwindels/memberlist branch, as it is 100% reproducible now.

[bwindels]

Indeed it was, so the original bug is likely about a different issue, but it's a good point that it might not be the eventid index that is faiing but rather the key because the eventIndex is somehow repeated.

[ptman]

I have no idea if related or not, but I get this on firefox:

"Sync failed because of StorageError: IDBRequest failed on hydrogen_session_8284842750974705.roomMembers: (name: AbortError) (code: 20) A request was aborted, for example through a call to IDBTransaction.abort."

Refreshing doesn't help. It seems I need to logout and log back in.

[bwindels]

@ptman what browser was that?

[ptman]

Firefox on linux

[bwindels]

I think you hit #180 rather @ptman. Thanks for reporting!