TypeError: Unknown chunk type 91 while reading record at

[jkapenieks]

Hi Acebase team, thanks for your great product.

I've run into a problem during my tests of Acebase server (acebase-server@1.4.0 running on Nodejs).
I've created a new database and have a script running that stores data in database.

After a while my server kept crashing and I noticed this kind of error (see below).
I'm not able to run any queries or delete data. This causes crash of my server script and this error appears in the server log.
NodeReader.getValue:child error: TypeError: Unknown chunk type 91 while reading record at "/items/0/3/16766808/item_data" @197,405
at NodeReader.readHeader (/......./node_modules/acebase/src/storage-acebase.js:2575:23)
at async NodeReader.getValue (/......./node_modules/acebase/src/storage-acebase.js:1843:13)
at async loadChildValue (/......./node_modules/acebase/src/storage-acebase.js:1919:37)
at async Promise.all (index 2)
at async NodeReader.getValue (/......./node_modules/acebase/src/storage-acebase.js:1963:21)
at async loadChildValue (/......./node_modules/acebase/src/storage-acebase.js:1919:37)

This is what I tried:
server.db.query('items')
.remove(() => {
// Old junk gone
});

Then I was trying to delete the node directly:

server.db.ref('items/0/3/16766808/item_data')
.remove()
.then(() => {

     }); 

This was the error:
/......./node_modules/acebase/src/storage-acebase.js:2575
throw new TypeError(Unknown chunk type ${type} while reading record at ${this.address});
^

TypeError: Unknown chunk type 91 while reading record at "/items/0/3/16766808/item_data" @197,405
at NodeReader.readHeader (/......./node_modules/acebase/src/storage-acebase.js:2575:23)
at async NodeReader.getAllocation (/......./node_modules/acebase/src/storage-acebase.js:1773:9)
at async /......./node_modules/acebase/src/storage-acebase.js:2688:36
at async Promise.all (index 0)
at async _mergeNode (/......./node_modules/acebase/src/storage-acebase.js:2701:5)
at async Object.write [as _customWriteFunction] (/......./node_modules/acebase/src/storage-acebase.js:1420:28)
at async AceBaseStorage._updateNode (/......./node_modules/acebase/src/storage-acebase.js:1433:26)
at async AceBaseStorage.setNode (/......./node_modules/acebase/src/storage-acebase.js:1349:9)
at async DataReference.set (/......./node_modules/acebase-core/dist/data-reference.js:158:13)
error: Forever detected script exited with code: 1

[jkapenieks]

Also, I noticed that my Acebase server script crashed when I was trying to navigate to the node: items/0/3/16766808/item_data on Acebase Web Manager.

[appy-one]

Thanks for reporting this. The best and quickest way for me to investigate this issue is having access to the actual data. If you can send me a copy of the data.db file through wetransfer.com, I will be able to poke into the binary data and see what's wrong and what might have caused it. You can send it to me@appy.one

Note that if your data contains personal and/or sensitive data, please create a copy of the db and anonymize your data as much as possible before sending. I will treat your data confidentially and will remove all copies from my system when done!

[appy-one]

Can you please reply

[appy-one]

I'm closing this issue. If you would like this fixed @jkapenieks, please reply. For anyone else running into this issue, please also let me know.

[neilsutcliffe]

Hi. I've ran into this as well. One of my paths refuses to read, which means I can't update or remove it.

Is there anyway to nuke it from orbit without reading it, or to read it and ignore errors

I can send you the database file, but at the moment, I'm just trying to recover my data from the last time I backed up.

[neilsutcliffe]

Tried... await acebaseLocal.ref('/image/7ddb9f615c2d9947be799b99d17ed59b11c61959[1]').set({})

GetTypeError: Unknown chunk type 112 while reading record at "/image/7ddb9f615c2d9947be799b99d17ed59b11c61959[1]" @51,340

I've also tried set([]) to the key directly, and also remove

[appy-one]

@neilsutcliffe Thanks for reporting this.

Are you running AceBase in a multi-process/threaded setup?

Also, I see you are directly updating an array entry, which is discouraged because of reasons described here: Using arrays. In most situations, it's better to use object collections.

I'll add a function to overwrite records today, so you will be able to fix this broken record.

[neilsutcliffe]

I'm running a next app with the built in api. So single threaded AFAIK.

In terms of how I was using arrays, I was setting the whole array at once, overwriting when updating.

path is like /image/7ddb9f615c2d9947be799b99d17ed59b11c61959

await acebaseFix.ref(path).set(images) // Update via overwrite
await acebaseFix.ref(path).set(newImages) // Create from scratch
await acebaseFix.ref(path).remove() // Delete

Oh, the image array members are structured...

export interface Image {
  width: number;
  height: number;
  img: string;
  original: string;
  url: string;
  cat?: string;
}

I'll give object collections a try, but I think the use case is fine, as I wasn't mutating or changing the array.

Thanks for getting back quickly!

[appy-one]

I've published version 1.22.2,which now supports node recovery. See #142 for more info
Let me know if that works.
Also, if you are sure you are not auto-scaling your app in the cloud, let me know if you find a reproducible way to "break" a node!

[appy-one]

Closing the issue again, feel free to reopen if this is still an issue!

[neilsutcliffe]

Hi. I keep on getting a different issue, but still general corruption. I'm having to back up my database any time I change it. Might look into collections, but I am not doing complex things with arrays...

Either way, I end up corrupting the database when working with Next.js. It could be the hot-reloading is creating multiple connections to the database, which causes this issue.

CorruptRecordError: CORRUPT RECORD: key "img" in "/image/77f9d37ee3c03f1d702a20b11e08cee22210ae5f[0]" (@22,369) refers to address @22,323 which was already used to read "/image/2ffeb7f6c7b5ca0606af6178a7009fb08d69ac4d". Recursive or repeated reading has been prevented.
    at new NodeReader (admin/node_modules/acebase/src/storage-acebase.js:2340:27)
    at loadChildValue (admin/node_modules/acebase/src/storage-acebase.js:2583:40) {
  record: [NodeAddress],
  key: 'img'
}

img in this case is a URL

I've taken to storing different tables in different databases allowing me to revert only one if needed.

[appy-one]

@neilsutcliffe You have to prevent multiple concurrent AceBase instances to the same database to prevent corruption. Reuse 1 single AceBase instance throughout your app to prevent this. Hot reloading should not be an issue, but having 2+ separate instances in your code both altering the same database will corrupt the db.

[neilsutcliffe]

@neilsutcliffe You have to prevent multiple concurrent AceBase instances to the same database to prevent corruption. Reuse 1 single AceBase instance throughout your app to prevent this. Hot reloading should not be an issue, but having 2+ separate instances in your code both altering the same database will corrupt the db.

It's 2 seperate instances with 2 different database files.

However it is reinitialising on hot reloading...

I think I might have to look more into how next.js does this and determines what to recompile. There has to be mechanisms to prevent it from rebuilding.