Tantivy fails with Failed purge deletes error

[winding-lines]

I am getting a failure starting my application

Failed purge deletes: Error(PathDoesNotExist("/SOME_PATH/c475e13ef3ca45128b1f8d9ee42fe994.term")

This is on OS X. My computer crashed recently. What's a good way to move forward here? Happy to help writing some code if that helps :)

[fulmicoton]

This might be an actual bug.

Which version of tantivy is this?
Can you give more details on what you were doing ?

Did you remove some files manually etc.

[fulmicoton]

Also is it something you can reproduce?

[fulmicoton]

I'm also interested in your meta.json if you still have it as well as logs.

[winding-lines]

@fulmicoton I did not delete files manually but my computer did crash.

I am using rev = "432d49d8147624", just some hash I used at that time.

In normal operation I am indexing html documents, the fields are the url, title and body. When adding a new document I delete the previous one, there is also a button to mark documents as private.

My implementation is dumb, I initialize searchers for every request. Here is my my meta.json file.

meta.json.zip

I do not have other logs.

[fulmicoton]

This sounds like a bug so your report is super helpful.

If you somehow found a way to reproduce, RUST_LOG=info will log all all of the updates on the segments sets (merges, commits, etc.).

[fulmicoton]

Oh wait... And you are saying that you get that on your application startup ?
I had totally missed that. This is very very peculiar.

The index itself is not something you can share, is it?
Can you join the results of ls ?

[winding-lines]

I don't think I can share the index right now but I've added a listing of the folder
ls.out.zip

I am trying to repro this, may need to wait until the weekend. I can send you an invite to the repo, it's on gitlab.

[fulmicoton]

@winding-lines

Can you retrace exactly what happened and the current state please ?

So far my understanding of the events :

• you created an index. Add and remove documents.
• your computer crashed once.
• you restarted your program
• Very quickly it panicked and died showing the following error : Failed purge deletes: Error(PathDoesNotExist("/SOME_PATH/c475e13ef3ca45128b1f8d9ee42fe994.term")
• If you restart your program right now, it works normally but has lost a bunch of its updates. (!?)

Can you especially confirm that last point?

[fulmicoton]

@winding-lines Thanks for the invite. I see the two projects listed, but I cannot see the source code somehow.

[winding-lines]

Sorry @fulmicoton, I am not very familiar with Gitlab, I've changed the settings some more let me know if things work better.

I will work over the weekend to try to isolate the problem in a smaller test case.

[winding-lines]

I had a bug in my start code and I was swallowing an Err() on process start.

DEBUG 2018-xx-xxT16:19:54Z: tantivy::directory::mmap_directory: Open Read "09dd5ceaa17542c09c42bc1f4cd58644.term"
ERROR 2018-xx-xxT16:19:54Z: error open index
ERROR 2018-xx-xxT16:19:54Z: caused by: path does not exist: '"/Users/SOME_PATH/09dd5ceaa17542c09c42bc1f4cd58644.term"'

Happy to add some more debugging statements if that helps :)

[fulmicoton]

@winding-lines Sorry I am still not entirely sure I understand what you experience and the current state of your index.

I still don't understand whether your index is currently in a corrupted state or not.

Ideally could you go through the list I wrote above, copy paste and edit it?

[winding-lines]

@fulmicoton I deleted the index for the time being and I am adding more logging should this happen at some other time. I think it's ok to close this issue since I cannot provide more information to help debug.

Sorry about the trouble.

Marius

[fulmicoton]

@winding-lines Reopening the issue.

If tantivy was working according to spec, your error message could only provoked manually by removing a file which you haven't done. There is most likely a severe issue here (thank you very much for reporting it).

If someone manages to reproduce it, please document it here.
Please set up and env_logger in your app to get tantivy logs.

[winding-lines]

Makes sense, I am also adding more logging and will try to reproduce on public pages. Marius ---

…

On Jun 22, 2018, at 6:23 PM, Paul Masurel ***@***.***> wrote: @winding-lines Reopening the issue. If tantivy was working according to spec, your error message could only provoked manually by removing a file which you haven't done. There is most likely a severe issue here (thank you very much for reporting it). If someone manages to reproduce it, please document it here. Please set up and env_logger in your app to get tantivy logs. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[fulmicoton]

Fix has been confirmed. Will be published as a hotfix in 0.6.1 today.