handle-search is getting stuck when using "containing..." #515

MarkA80 · 2021-01-24T01:24:17Z

Hi,
I am using wiki-serve 3.1.2-4 for windows with the latest .zim.

When using the search bar and selecting "containing..."
Doing the verbose, it gets stuck on the handle search.
Is it possible to disable this feature with a start option!
The normal search where the suggestions are popping up is working.

I tried out the kiwix reader on mac and it seems to have a similar issue when doing several searches, that it crashes.
The plugin for the webbrowser seems to be fine.

My work around is to stop the process every 30 seconds and restart it.

kelson42 · 2021-01-24T13:00:18Z

Which ZIM file exactly? Which search pattern exactly?

MarkA80 · 2021-01-25T11:11:01Z

I thought all of the ones I have installed are affected, WikiMed, Chemistry, Mathematics
and any kind of pattern, as soon as I click "containing..."
But just found out that the Wiki for Schools, the full text is working as it should.
I have downloaded all, except the WikiMed through torrent and through slow satellite internet.
Could the .zim files be corrupted? Other than the full text search not working, the content is showing up when doing the normal search and selecting the link or navigating through the portals and wiki links.

OscarL · 2021-01-31T16:40:05Z

I'm having the same problem on Win10 64 with kiwix-serve version 3.1.2.

Trying to open http://localhost:8080/search?content=archlinux_en_all_maxi_2020-11&pattern=test ends with the following console output, and I have to CTRL-C to end the process (Windows shows a "Debug/Close Program" dialog, but I have no debugger, so it just ends).

D:\Kiwix>kiwix-serve-new -v --port 8080 --library library.xml
======================
Requesting :
full_url  : /search
method    : GET (0)
version   : HTTP/1.1
request#  : 0
headers   :
 - accept : 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
 - accept-encoding : 'gzip, deflate'
 - accept-language : 'es-AR'
 - connection : 'Keep-Alive'
 - host : 'localhost:8080'
 - upgrade-insecure-requests : '1'
 - user-agent : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18363'
arguments :
 - content : 'archlinux_en_all_maxi_2020-11'
 - pattern : 'test'
Parsed :
full_url: /search
url   : /search
acceptEncodingDeflate : 1
has_range : 0
is_valid_url : 1
.............
** running handle_search
Performing query `test'
Setup queryparser using language eng
Parsed query 'test' to Query(test@1)

D:\Kiwix>

If I try that search on and old zim I have around (arduino.stackexchange.com_en_all_2018-03), opening the URL "http://localhost:8080/search?content=arduino.stackexchange.com_en_all_2018-03&pattern=test, it works OK.

In fact, it works with: wikipedia_en_all_maxi_2018-10 and wikipedia_es_all_maxi_2019-09.zim too, But fails again with wikipedia_es_all_mini_2021-01.zim

kelson42 · 2021-02-04T09:57:49Z

I confirm, kiwix-serve get stuck... but does not crash and does not returns anything.

jamalex · 2021-04-12T18:13:47Z

Observed this here as well. Win10, with kiwix-serve.exe from kiwix-tools_win-i686-3.1.2-4.zip.

Run: kiwix-serve.exe --port=7070 wikipedia_en_100_maxi_2021-03.zim

Loads fine in the browser, but full-text search hangs indefinitely, while trying to load URL:
http://127.0.0.1:7070/search?content=wikipedia_en_100_maxi_2021-03&pattern=animal+
(with no output on the command line)

kelson42 · 2021-05-02T11:18:55Z

@MananJethwani @mgautierfr This bug HAS to be fixed before next release.

mgautierfr · 2021-05-28T14:13:49Z

After investigation, there are several points :

The bug should appears with libzim 6.3.0 AND zim file using zstd compression.
We are compressing our content with level 22, which imply a windows (to search occurrences) of (1^27 Bytes, 128Mb). So at decompression level, we need to allocate such memory.
On top of that, libzim 6.3.0 do partial decompression. We do not decompress all the cluster but just enough to get the wanted blob. To avoid decompress several time the same value, we keep in cache the decompression context and so we keep alive a 128Mb buffer per clusters.

With a default cache of 16 cluster, this means we allocate 2Gb.
On linux, lazy allocation is made, the 2Gb are not really allocated and we really use want needed (the clusters are about 1Mb by default, so a total of around 16Mb).
But on windows, this is real allocation and we run out of memory (on top of the 2Gb, we need 128Mb of continuous memory).
If you add a bug in libzim itself where we are not checking return value of the decompression step in a loop, we have this bug, we are stuck in a infinite loop.

What to do :

Fix the decompression loop to detect the problem. At least we will have a 500 error code on kiwix-serve and not a infinite loop. (But it will not reduce the memory usage)
Compress less our content. Level 19 need a 8Mb buffer for almost the same compression ratio.

A workaround for this bug is to reduce the cluster cache size by setting the env variable ZIM_CLUSTERCACHE. (ZIM_CLUSTERCACHE=2)

kelson42 · 2021-05-28T14:27:52Z

Ping @veloman-yunkan in case you miss this discussion.

kelson42 · 2021-05-28T14:33:09Z

@mgautierfr Thank you for the investigation and the analysis. Your two proposal look good to me. This should be quickly implemented and one time merged, we should backport them and release a libzim 6.3.1. We should as well quickly if this resolve as well #513

If somehow `Decoder::stream_run_encode` fails, `m_decoderState.avail_out` will never be updated and we are in a infinite loop. We must exit the loop at a moment. Raising a exception may crash the user code if it doesn't catch it. For kiwix-serve, we will return a 500 error code but we will still be running. See kiwix/libkiwix#515

The max level (22) set a huge windowSize (128Mb). It if far too much and we don't even need it (most of our compressed content is about 1Mb). By setting a level of 19, the windowSize is 8Mb and the compression ratio is relativly equivalent. From lzbench https://github.com/inikep/lzbench the ratio is 24.88 for level 22 and 25.33 for level 18. See kiwix/libkiwix#515

kelson42 · 2021-05-28T16:19:27Z

@mgautierfr With the fix how many zim can beliaded in parallel in a RPI W Zero?

The max level (22) set a huge windowSize (128Mb). It if far too much and we don't even need it (most of our compressed content is about 1Mb). By setting a level of 19, the windowSize is 8Mb and the compression ratio is relativly equivalent. From lzbench https://github.com/inikep/lzbench the ratio is 24.88 for level 22 and 25.33 for level 18. See kiwix/libkiwix#515

`CompStatus::OTHER` is not really used and is mainly used as a error code (if used). Let's remove it and directly throw a exception instead of returning it. This commit make the ZSTD/LZMA error correctly handled (at least not silently ignored) during (de)compression. See kiwix/libkiwix#515

We must handle exception raised in the callable in `getOrPut`. If not, we would have a broken promise in the cache. It is better to have no promise in the cache at all if we fail to create the value. (The concurent cache user as no notion of promise). And sometime, the creation of the value is temporarly failing. For exemple, in kiwix/libkiwix#515, if some memory is free, we would be able to create a cluster even if we failed the last time.

`CompStatus::OTHER` is not really used and is mainly used as a error code (if used). Let's remove it and directly throw a exception instead of returning it. This commit make the ZSTD/LZMA error correctly handled (at least not silently ignored) during (de)compression. See kiwix/libkiwix#515

We must handle exception raised in the callable in `getOrPut`. If not, we would have a broken promise in the cache. It is better to have no promise in the cache at all if we fail to create the value. (The concurent cache user as no notion of promise). And sometime, the creation of the value is temporarly failing. For exemple, in kiwix/libkiwix#515, if some memory is free, we would be able to create a cluster even if we failed the last time.

`CompStatus::OTHER` is not really used and is mainly used as a error code (if used). Let's remove it and directly throw a exception instead of returning it. This commit make the ZSTD/LZMA error correctly handled (at least not silently ignored) during (de)compression. See kiwix/libkiwix#515

We must handle exception raised in the callable in `getOrPut`. If not, we would have a broken promise in the cache. It is better to have no promise in the cache at all if we fail to create the value. (The concurent cache user as no notion of promise). And sometime, the creation of the value is temporarly failing. For exemple, in kiwix/libkiwix#515, if some memory is free, we would be able to create a cluster even if we failed the last time.

`CompStatus::OTHER` is not really used and is mainly used as a error code (if used). Let's remove it and directly throw a exception instead of returning it. This commit make the ZSTD/LZMA error correctly handled (at least not silently ignored) during (de)compression. See kiwix/libkiwix#515

We must handle exception raised in the callable in `getOrPut`. If not, we would have a broken promise in the cache. It is better to have no promise in the cache at all if we fail to create the value. (The concurent cache user as no notion of promise). And sometime, the creation of the value is temporarly failing. For exemple, in kiwix/libkiwix#515, if some memory is free, we would be able to create a cluster even if we failed the last time.

The max level (22) set a huge windowSize (128Mb). It if far too much and we don't even need it (most of our compressed content is about 1Mb). By setting a level of 19, the windowSize is 8Mb and the compression ratio is relativly equivalent. From lzbench https://github.com/inikep/lzbench the ratio is 24.88 for level 22 and 25.33 for level 18. See kiwix/libkiwix#515

mgautierfr · 2021-06-03T14:28:32Z

Please test with binaries in http://tmp.kiwix.org/kiwix-tools_3.1.2-5/

kelson42 · 2021-06-03T14:51:03Z

@jamalex @MarkA80 @OscarL Let us know quickly - if possible - about the result of your test. We would like to make a new release only with this fix.

jamalex · 2021-06-03T22:10:16Z

I can confirm that kiwix-tools_win-i686-3.1.2-5-rc1/kiwix-serve.exe responds quickly and reliably, when I try it with the case I mentioned above that was previously hanging.

kelson42 · 2021-06-04T06:38:04Z

@jamalex Thank you for testing again. An official new maintenance release of kiwix-tools will be done in the next days. Closing the ticket.

OscarL · 2021-06-04T19:59:32Z

@kelson42: Just to add another data point (sorry for not replying sooner)... With the test binaries (from kiwix-tools_win-i686-3.1.2-5-rc1.zip), search using "contains..." works without getting stuck! Which means I'm now able to use this version (with newer .zim files!).

Thanks to all of you for your work. It is appreciated!

kelson42 self-assigned this Jan 24, 2021

kelson42 removed their assignment Feb 4, 2021

kelson42 pinned this issue May 2, 2021

kelson42 transferred this issue from kiwix/kiwix-tools May 13, 2021

kelson42 added bug kiwix-serve windows labels May 13, 2021

kelson42 added this to the 10.0.0 milestone May 13, 2021

mgautierfr self-assigned this May 27, 2021

kelson42 mentioned this issue May 28, 2021

Kiwix Serve freezing on RPI with a lot of ZIM files #513

Closed

mgautierfr mentioned this issue May 28, 2021

Better decompression buffer size for small content and high compression level. facebook/zstd#2688

Closed

mgautierfr mentioned this issue May 28, 2021

Check decompression status in DecoderStreamReader. openzim/libzim#568

Merged

mgautierfr mentioned this issue May 28, 2021

Lower zstd compression level. openzim/libzim#569

Merged

This was referenced May 29, 2021

Clicking quickly many times on random crashes kiwix-serve for Windows #512

Closed

Larger .zim files not searchable with Kiwix-serve on RPI W #542

Closed

holta mentioned this issue Jun 3, 2021

Test kiwix-tools 3.1.2-5 RC1 for armhf iiab/iiab#2810

Closed

kelson42 closed this as completed Jun 4, 2021

holta mentioned this issue Jun 10, 2021

kiwix-tools 3.1.2-4 -> 3.1.2-5 iiab/iiab#2815

Merged

mgautierfr mentioned this issue Oct 25, 2021

Internal server error: out of memory kiwix/kiwix-tools#492

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

handle-search is getting stuck when using "containing..." #515

handle-search is getting stuck when using "containing..." #515

MarkA80 commented Jan 24, 2021

kelson42 commented Jan 24, 2021

MarkA80 commented Jan 25, 2021

OscarL commented Jan 31, 2021 •

edited

Loading

kelson42 commented Feb 4, 2021

jamalex commented Apr 12, 2021

kelson42 commented May 2, 2021

mgautierfr commented May 28, 2021

kelson42 commented May 28, 2021

kelson42 commented May 28, 2021

kelson42 commented May 28, 2021

mgautierfr commented Jun 3, 2021

kelson42 commented Jun 3, 2021

jamalex commented Jun 3, 2021

kelson42 commented Jun 4, 2021

OscarL commented Jun 4, 2021

handle-search is getting stuck when using "containing..." #515

handle-search is getting stuck when using "containing..." #515

Comments

MarkA80 commented Jan 24, 2021

kelson42 commented Jan 24, 2021

MarkA80 commented Jan 25, 2021

OscarL commented Jan 31, 2021 • edited Loading

kelson42 commented Feb 4, 2021

jamalex commented Apr 12, 2021

kelson42 commented May 2, 2021

mgautierfr commented May 28, 2021

kelson42 commented May 28, 2021

kelson42 commented May 28, 2021

kelson42 commented May 28, 2021

mgautierfr commented Jun 3, 2021

kelson42 commented Jun 3, 2021

jamalex commented Jun 3, 2021

kelson42 commented Jun 4, 2021

OscarL commented Jun 4, 2021

OscarL commented Jan 31, 2021 •

edited

Loading