Lowercase search does not find non-ASCII uppercase pages and vice versa

[apolukhin]

Describe the bug
Searching "привет" while the page title is "Привет" does not work.

Expected behavior
For ASCII everything works fine: searching for "faq" finds page "FAQ". The same behavior expected for non ASCII pages.

Screenshots

To Reproduce

1. Make some markdown pages starting with # Привет, # Основные сведения, # прочее and # Введение.
2. Generate docs from those pages using the SEARCHENGINE=YES and SERVER_BASED_SEARCH=NO
3. Search for при, ПРИВ, основ, ПРОЧЕЕ, вВеден

Version
1.9.1 and trunk.

[albert-github]

Recently I've see a fix for a similar issue #5263 that might also might fix your problem (although you mentioned also trunk with the problem), though it is hard to tell without an example (I don't have a good example at hand with Cyrillic search terms).

• Please specify the trunk version you used (complete information of doxygen -v).
• Can you please attach a, small, self contained example (source+configuration file in a tar or zip) that allows us to reproduce the problem? Please don't add external links as they might not be persistent.

[apolukhin]

doxygen -v
1.9.2 (46ffd77e8d3c15244732128caace858f2aa38d73)

Searching for "ОСН", "ПРИ", "При" and "проч" works (was fixed in #5263, thanks!)
Searching for "при", "вве", "ПРОЧ", "осн", does not work

Attaching a self contained example, all the info is duplicated inside in Doxygen config:
doxygen_sample.zip

[albert-github]

It was at first a bit confusing as the output language was still English, but the problem here lies with the names of the sections in the related pages.

[albert-github]

The problem looks like a bit more a fundamental problem when writing (a.o. search/all*.js, search/pages*.js, search/searchdata.js) it looks like the translation to lowercase is not done as it is done for the Latin alphabet / ASCII

[albert-github]

I've just pushed a proposed patch, pull request #8409

[doxygen]

@apolukhin Please verify if commit a4ecbee fixes the problem for you.

[albert-github]

As far as I can see it does not work.

Example: example.tar.gz

Here we have the source to generate the html pages and the directories:

• html_mine the approach as I proposed in issue #8375 Lowercase search does not find non-ASCII uppercase pages and vice versa #8409
• html_c a run on Cygwin
• html_w a run on Windows

When going to the related pages (for easy cut and paste) an cutting the text and pasting the text into the search bar:

• Основные сведения In my approach shows the reference and in the Cygwin and Windows version gives "No Matches", first as stripping down to the space it shows a match on Cygwin but on Windows still not.
  See the subtle difference (space versus _20):

  diff -r html_c/search/pages_1.js html_mine/search/pages_1.js
  3c3
  <   ['основные сведения_5',['Основные сведения',['../md_test2.html',1,'']]]
  ---
  >   ['основные_20сведения_5',['Основные сведения',['../md_test2.html',1,'']]]
  

• прочее in my approach and Cygwin this looks OK, for Windows I get in the window even a file not found reference.
  see:

  diff -b -w -r html_mine/search/searchdata.js html_w/search/searchdata.js
  3,4c3,4
  <   0: "воп",
  <   1: "воп"
  ---
  >   0: "вопп",
  >   1: "вопп"
  

[doxygen]

@albert-github Fixed two issues:

• Visual Studio interprets u8"..." string literals in the locale encoding (so not UTF-8!) which caused the scrambled output. Can be fixed by using the /utf-8 compiler option, but since other compilers may have similar issues I decided to put the exact byte encoding in the caseconvert.h file instead like you did.
• I missed the escaping of non-identifier characters in function searchId()

Let me know if you see more issues.

[albert-github]

• The good old locale, always giving problems ....
  Looks like to be working now. I tested it with a Japanese string as well.
• Escaping of non-identifier characters
  Looks like to be working now

[apolukhin]

@apolukhin Please verify if commit a4ecbee fixes the problem for you.

@doxygen yep, latest master works like a charm.

Many thanks!

[doxygen]

This issue was previously marked 'fixed but not released',
which means it should be fixed in doxygen version 1.9.2.
Please verify if this is indeed the case. Reopen the
issue if you think it is not fixed and please include any additional information
that you think can be relevant (preferably in the form of a self-contained example).