DEV ERROR: index should now be at the known end of the data

[ahdat81]

not exactly sure what happened, but suddenly ran into this error: DEV ERROR: index should now be at the known end of the data, when trying to get an index.

Is there a way to fix the db? perhaps delete the index and recreate it? Just saw a post about corrupting the db, that's probably going to be important

[Mitsichury]

Hi !
Do you wait for the database to be ready before creating the indexes ?

[ahdat81]

I was partially able to fix it. On a whim i removed the index files from the db folder and the problem went away. But on inspection one of the nodes is set to {}. It could be my code, but it also might be some conflict when i was testing/debugging with the cache, curser, multiple coputers, etc...

Side note i noticed theres a indexes.delete() function. is there a meaningful reason to use this, rather than just delete the file? I tried pointing it to the index files but says it cant be found.

[appy-one]

Corruption of index files should not happen if they are accessed from 1 process at the same time, or if you have proper IPC settings configured. Removing an index can be done through the filesystem as long as AceBase is not running. They will be recreated upon restart if you still have the db.indexes.create in your code.

If you run into this more frequently and are able to reproduce, please create an issue for it!

[ahdat81]

this was my first time running into this so not frequent. but will def post if i can reproduce

[Mitsichury]

Hi :)
I've been able to reproduce it too
Same process, but I use a setInterval that is cleaning old values
If the server have a request (and query data from db) during the cleaning the foolowing error is throw:

DEV ERROR: index should now be at the known end of the data
"XXX.idx" timed out! lock.release() was not called in a timely fashion

I'll try to create a detailled issue

[Mitsichury]

Hi everyone !

I've been trying to reproduce a consistent beahavior but it's difficult, I don't know if can write a UT for this case :(
What I suppose is when I delete an indexed value, indexes are not well updated.

Then the index is rebuilt and the database runs again. But the next queries will be likely to fail with the index error location :/

I did not succeed to reproduce it when adding data.

@appy-one Could you recommand me a way to clean periodically old data safely, I'm currently using setInterval but I don't know if it's a safe way :D

Thank you,
Mitsi.

[ahdat81]

I noticed deleting nodes can be 'dangerous' since it invovles so many pieces (db, cache, live objects, other users, etc...). maybe appy has a better way, but i've been flagging my nodes for deletion. Then actually perform the delete when the db is restarted, everyone is disconnected, immediately after the db is ready.

[appy-one]

Thanks for your info @Mitsichury and @ahdat81, I'll further investigate the effects of deleting indexed nodes. I'll keep you posted.

[appy-one]

@Mitsichury @ahdat81 I've improved query.remove performance (drastically) and fixed a few issues in latest packages. Let me know if this improves things

[ahdat81]

@appy-one I updated to the latest build and was having problems with the cache (last updated maybe a month or two ago). Everything works initially (cache is cleared). But when i log back in, during my initialization process (where i create about a dozen proxies) somewhere it the middle the db hangs. DIdnt' really see anything in the logs, just connection timeout errors. I have to restart the db and clear my cache. But with the cache disabled when creating a db connection, everything appears to be working fine.