Sort with collator #3115

moisesbr-dw · 2020-06-01T20:23:31Z

When intl extension is available, all alphabetical sorts are done using a collator. Otherwise, primitive PHP functions are called.

The collator is created using the locale given in $conf['lang']. It always uses case insensitive "natural" ordering in its collation. The fallback solution uses the primitive PHP functions that return almost the same results when the input is text with only [A-Za-z0-9] characters.

All those screens will be properly sorted when intl extension is configured in php.ini:

Sitemap (Index)
Search
- Matching pagenames
- Fulltext results [added]
- namespace selector below search field
Quick search (when typing)
Backlinks
All media lists (including fullscreen)
- Media backlinks (View tab in the 3rd column of Media Manager)
Internal link suggestion when editing a page
Administration
Access Control List Management
- tree list of namespaces and pages
- drop-down list with users and groups
- rule list by page/namespace (main sort)
- rule list by user/group (secondary sort) [added]
Authorization plugins
- group sort in authplain [added]
List of all pages in Remote API [added]
OpenSearch suggestions
Command line tool wantedpages.php

Future maintenance (e.g. plugins) will be very easy. Alphabetical comparisons and sorts should be changed the following way:

from strcmp(), strcasecmp(), strnatcmp() and strnatcasecmp() to Sort::strcmp()
from sort() to Sort::sort()
from ksort() to Sort::ksort()
from asort(), natsort() and natcasesort() to Sort::asort() or Sort::asortFN()

Closes #2037 (collation part).

New file "sort.php" for sort functions. Inclusion of "sort.php" in the bootstrap ("load.php"). Reimplementation of natsort() using collator. Fix of search() function in "search.php". Functions that call search() and now have the expected results: Ajax.php:188, callMedians(): Return subnamespaces for the Mediamanager Ajax.php:322, callIndex(): Return sub index for index view Ajax.php:392, callLinkwiz(): List matching namespaces and pages for the link wizard html.php:908, html_index(): Display page index media.php:713, media_filelist(): List all files in a given Media namespace media.php:1528, media_searchlist(): List all files found by the search request media.php:1966, media_nstree(): Build a tree outline of available media namespaces Remote\ApiCore.php:349, readNamespace(): List all pages in the given namespace (and below) Remote\ApiCore.php:432, listAttachments(): List all media files Screens fixed: - Index - All media lists (including fullscreen) - Internal link suggestion when editing a page

Reimplementation of strcmp() using collator. Fix of ft_pagesorter() function in "fulltext.php". ft_pagesorter() is used by _ft_pageLookup(), which is used by ft_pageLookup(). Functions that call ft_pageLookup() and now have the expected results: Action\Search.php:69, execute(): run the search Ajax.php:50, callQsearch(): Searches for matching pagenames Ajax.php:352, callLinkwiz(): List matching namespaces and pages for the link wizard Screens fixed: - Search (list of Matching pagenames) - Quick search (when typing) - Internal link suggestion when editing a page (when nothing was typed)

Reimplementation of sort() using collator. Fix of ft_backlinks() and ft_mediause() function in "fulltext.php". Functions that call ft_backlinks() and now have the expected results: html.php:1085, html_backlinks(): display backlinks Remote\ApiCore.php:456, listBackLinks(): Return a list of backlinks Functions that call ft_mediause() and now have the expected results: media.php:222, media_inuse(): Convenience function to check if a media file is still in use media.php:1199, media_details(): Prints mediafile tags Screens fixed: - Backlinks - List of pages that references a media file (View tab in the 3rd column of Media Manager)

Fix of callSuggestions() function in "Ajax.php". Screens fixed: - OpenSearch suggestions: /lib/exe/ajax.php?call=suggestions&q={query}

Reimplementation of ksort() using collator. Fix of getAdditionalNamespacesFromResults() function in "Ui/Search.php". Screens fixed: - Search (namespace selector below search field)

Reimplementation of strcasecmp() using collator. It uses the same implementation already made for strcmp(). Fix of menuSort() function in "Ui/Admin.php". Screens fixed: - Administration

Fix of sort_search_fulltext() function in "search.php". This function does not seem to be used anywhere.

Debug messages added.

Name changed from natural_sort() to sort_filenames(). Fixed behavior without collator: page name sorting reflected in filename array. Fix of search() function in "search.php". Functions that call search() and now have the expected results: Ajax.php:188, callMedians(): Return subnamespaces for the Mediamanager Ajax.php:322, callIndex(): Return sub index for index view Ajax.php:392, callLinkwiz(): List matching namespaces and pages for the link wizard html.php:908, html_index(): Display page index media.php:713, media_filelist(): List all files in a given Media namespace media.php:1528, media_searchlist(): List all files found by the search request media.php:1966, media_nstree(): Build a tree outline of available media namespaces Remote\ApiCore.php:349, readNamespace(): List all pages in the given namespace (and below) Remote\ApiCore.php:432, listAttachments(): List all media files Screens fixed: - Index - All media lists (including fullscreen) - Internal link suggestion when editing a page

Name changed from strcompare() to compare(). Fix of ft_pagesorter() function in "fulltext.php". ft_pagesorter() is used by _ft_pageLookup(), which is used by ft_pageLookup(). Functions that call ft_pageLookup() and now have the expected results: Action\Search.php:69, execute(): run the search Ajax.php:50, callQsearch(): Searches for matching pagenames Ajax.php:352, callLinkwiz(): List matching namespaces and pages for the link wizard Screens fixed: - Search (list of matching pagenames) - Quick search (when typing) - Internal link suggestion when editing a page (when nothing was typed) Fix of menuSort() function in "Ui/Admin.php". Screens fixed: - Administration Fix of sort_search_fulltext() function in "search.php". This function does not seem to be used anywhere.

Name changed from sort_pages() to sort_pagenames(). Fixed behavior without collator: sort() with flags SORT_NATURAL and SORT_FLAG_CASE. Fix of ft_backlinks() and ft_mediause() function in "fulltext.php". Functions that call ft_backlinks() and now have the expected results: html.php:1085, html_backlinks(): display backlinks Remote\ApiCore.php:456, listBackLinks(): Return a list of backlinks Functions that call ft_mediause() and now have the expected results: media.php:222, media_inuse(): Convenience function to check if a media file is still in use media.php:1199, media_details(): Prints mediafile tags Screens fixed: - Backlinks - List of pages that references a media file (View tab in the 3rd column of Media Manager) Fix of callSuggestions() function in "Ajax.php". Screens fixed: - OpenSearch suggestions: /lib/exe/ajax.php?call=suggestions&q={query}

Fixed behavior without collator: ksort() with flags SORT_NATURAL and SORT_FLAG_CASE. Fix of getAdditionalNamespacesFromResults() function in "Ui/Search.php". Screens fixed: - Search (namespace selector below search field)

Fix of functions _sort_filenames_without_collator() and compare(). Updated documentation.

Better naming of functions to help future maintenance. - compare() renamed to intl_strcmp() - sort_pagenames() renamed to intl_sort() - sort_keys() renamed to intl_ksort() - new function intl_asort() - sort_filenames() renamed to intl_asortFN() Updated documentation.

All sortable lists: - tree list of namespaces and pages - drop-down list with users and groups - rule list by page/namespace (main sort) - [enhancement] rule list by user/group (secondary sort)

[enhancement] group sorting in authplain

As the $data sort order has been fixed in search() (see "search.php"), utf8_encodeFN() must not be used in the comparison.

getPagelist(), getAttachments() and getBackLinks() provided sorted results, but listPages() did not. [bug fix] Added missing 'doc' keys in lines 145 and 149.

_ft_pageLookup() provided sorted results, but _ft_pageSearch() did not.

splitbrain

This looks good except for my one comment about wrapping it in a class. @moisesbr-dw can you change that? Or shall I do it?

inc/sort.php

splitbrain · 2020-08-10T14:31:07Z

Can we change the tests in a way that they use a dataProvider or load different test data from files, so we can setup test in various languages?

moisesbr-dw · 2020-08-11T04:05:38Z

This looks good except for my one comment about wrapping it in a class. @moisesbr-dw can you change that? Or shall I do it?

Sorry @splitbrain , I'll be very busy in the next days.
I'm glad you've found the changes worth merging!

moisesbr-dw · 2020-08-11T04:14:13Z

Can we change the tests in a way that they use a dataProvider or load different test data from files, so we can setup test in various languages?

Only function test_intl_strcmp() uses a dataProvider.
This could certainly be extended to the other tests.
Unfortunately I won't be able to change the tests as soon as I would want.

The proper use of data providers now make it much easier to add addtional languages to the test

Now the use of the intl extension can be turned off, allowing for easy testing of the fallback. The test now inherits from the collator test so we avoid too much duplicate code

Changes for the collator-based sort: sort functions as static methods; better testing

Many minor details and use of Sort::xyz() instead of intl_xyz() in files outside the "inc" folder.

moisesbr-dw · 2020-08-16T21:58:06Z

@splitbrain All tests are passing. Just to be sure, I'll pass through all screens listed in the first message before considering the PR done.

Collations for Portuguese and Spanish; examples for Portuguese; better comments

Correction for German collation; examples for German and Spanish; much better comments

moisesbr-dw · 2020-08-20T09:19:23Z

Now I consider the test classes fully functional, testing everything in the Esperanto, German, Portuguese and Spanish collators.
I've written extensive comments to guide anyone interested in testing a collator for another language.

I still have to verify the screens as I've mentioned before.
I hope I'll manage to do that on the weekend.

@splitbrain could you please check the German data in the tests (last commit)?

@lmachucab could you please do the same with the Spanish data?

moisesbr-dw · 2020-08-23T23:16:41Z

Screens tested.
From my part, all set and done, @splitbrain !

splitbrain · 2020-08-26T08:06:42Z

Merged. Excellent work here! Thanks a lot!

Klap-in · 2020-08-26T09:11:34Z

Thanks for this work!
Could you update the function names in the start post please? Then I could use that for documentation updates at other places.

moisesbr-dw · 2020-08-26T16:54:53Z

Thanks for this work!
Could you update the function names in the start post please? Then I could use that for documentation updates at other places.

Done!

moisesbr-dw · 2020-08-26T16:56:28Z

Maybe a devel:sort page could be inserted somewhere in the Development Manual.

moisesbr-dw · 2020-08-26T18:26:59Z

@Klap-in please take a look at the files attached in this comment, as they provide valuable information.

lmachucab · 2020-08-30T17:01:11Z

I'll try to get to this when I have some extra time. Haven't really touched anything DW-related in a few weeks.

lmachucab · 2020-08-30T17:01:20Z

I'll try to get to this when I have some extra time. Haven't really touched anything DW-related in a few weeks.

moisesbr-dw added 30 commits May 2, 2020 23:40

Reimplementation of sort() [continued]

00d9ae5

Fix of callSuggestions() function in "Ajax.php". Screens fixed: - OpenSearch suggestions: /lib/exe/ajax.php?call=suggestions&q={query}

Reimplementation of ksort()

5f10907

Reimplementation of ksort() using collator. Fix of getAdditionalNamespacesFromResults() function in "Ui/Search.php". Screens fixed: - Search (namespace selector below search field)

Reimplementation of strcasecmp()

49097b7

Reimplementation of strcasecmp() using collator. It uses the same implementation already made for strcmp(). Fix of menuSort() function in "Ui/Admin.php". Screens fixed: - Administration

Reimplementation of strcmp() [continued]

0f3b1a8

Fix of sort_search_fulltext() function in "search.php". This function does not seem to be used anywhere.

Cleanup of function _init_collator()

ea0a567

Debug messages added.

Cleanup of function sort_keys()

3d31297

Fixed behavior without collator: ksort() with flags SORT_NATURAL and SORT_FLAG_CASE. Fix of getAdditionalNamespacesFromResults() function in "Ui/Search.php". Screens fixed: - Search (namespace selector below search field)

strnatcasecmp() is the correct fallback for $collator->compare()

0485973

Fix of functions _sort_filenames_without_collator() and compare(). Updated documentation.

Debug message in one line

651e547

Rearrangement of "sort.php" in the bootstrap ("load.php")

6de61e3

Rearrangement of function order in "sort.php"

f3a130c

Documentation

0577ab0

Collator not visible on global scope

41161b4

Affected files outside doc comments

9b065b6

Access Control List Management

44eedca

All sortable lists: - tree list of namespaces and pages - drop-down list with users and groups - rule list by page/namespace (main sort) - [enhancement] rule list by user/group (secondary sort)

Command line tool wantedpages.php

d00f852

Authorization plugins

8371b18

[enhancement] group sorting in authplain

Insertion of current namespace in Media Manager tree list

aa422a5

As the $data sort order has been fixed in search() (see "search.php"), utf8_encodeFN() must not be used in the comparison.

Page sort added in listPages()

86125dd

getPagelist(), getAttachments() and getBackLinks() provided sorted results, but listPages() did not. [bug fix] Added missing 'doc' keys in lines 145 and 149.

Page sort added in _ft_pageSearch()

06281c9

_ft_pageLookup() provided sorted results, but _ft_pageSearch() did not.

Comments with altered files removed

f854b31

Documentation update

169d126

Test class for sort with collator

30ad1f2

Test class for sort without collator

0384d31

splitbrain requested changes Aug 10, 2020

View reviewed changes

inc/sort.php Outdated Show resolved Hide resolved

splitbrain added 3 commits August 11, 2020 14:06

wrap sorting functions into their own class

2d85e84

cleanup for collator tests

5d9cb4e

The proper use of data providers now make it much easier to add addtional languages to the test

test the collator fallback always

f9aa34a

Now the use of the intl extension can be turned off, allowing for easy testing of the fallback. The test now inherits from the collator test so we avoid too much duplicate code

splitbrain mentioned this pull request Aug 11, 2020

suggested changes for the collator based sort moisesbr-dw/dokuwiki#1

Closed

moisesbr-dw added 2 commits August 16, 2020 03:22

Merge pull request #2 from splitbrain/pr-3115

ec16ab5

Changes for the collator-based sort: sort functions as static methods; better testing

Changes according to revisions in #2

0489c64

Many minor details and use of Sort::xyz() instead of intl_xyz() in files outside the "inc" folder.

moisesbr-dw added 2 commits August 17, 2020 09:08

Tests for Portuguese and Spanish

d0fecb1

Collations for Portuguese and Spanish; examples for Portuguese; better comments

Thorough tests for EO, DE, PT and ES

242e693

Correction for German collation; examples for German and Spanish; much better comments

splitbrain merged commit d267a3c into dokuwiki:master Aug 26, 2020

This was referenced Aug 26, 2020

Norwegian translation should use the correct language code #3242

Open

DokuWiki should do search ignoring accents (diacritics) #3243

Open

moisesbr-dw mentioned this pull request Nov 13, 2020

Automatic verification of language codes for new translations #3335

Open

Klap-in mentioned this pull request Jul 11, 2021

Country-specific characters are not sorted correctly samuelet/indexmenu#75

Closed

ssahara added a commit to ssahara/dokuwiki that referenced this pull request Nov 29, 2021

catch up dokuwiki#3115 Sort with collator

a02395a

Klap-in mentioned this pull request Oct 20, 2022

qsearch (pagename quicksearch) - bug when search word starts with lower case ü umlaut #3808

Closed

Klap-in mentioned this pull request Aug 25, 2023

Reverse sort with Collator #4041

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sort with collator #3115

Sort with collator #3115

moisesbr-dw commented Jun 1, 2020 •

edited

splitbrain left a comment

splitbrain commented Aug 10, 2020

moisesbr-dw commented Aug 11, 2020

moisesbr-dw commented Aug 11, 2020

moisesbr-dw commented Aug 16, 2020

moisesbr-dw commented Aug 20, 2020

moisesbr-dw commented Aug 23, 2020

splitbrain commented Aug 26, 2020

Klap-in commented Aug 26, 2020 •

edited

moisesbr-dw commented Aug 26, 2020

moisesbr-dw commented Aug 26, 2020

moisesbr-dw commented Aug 26, 2020

lmachucab commented Aug 30, 2020

lmachucab commented Aug 30, 2020

Sort with collator #3115

Sort with collator #3115

Conversation

moisesbr-dw commented Jun 1, 2020 • edited

splitbrain left a comment

Choose a reason for hiding this comment

splitbrain commented Aug 10, 2020

moisesbr-dw commented Aug 11, 2020

moisesbr-dw commented Aug 11, 2020

moisesbr-dw commented Aug 16, 2020

moisesbr-dw commented Aug 20, 2020

moisesbr-dw commented Aug 23, 2020

splitbrain commented Aug 26, 2020

Klap-in commented Aug 26, 2020 • edited

moisesbr-dw commented Aug 26, 2020

moisesbr-dw commented Aug 26, 2020

moisesbr-dw commented Aug 26, 2020

lmachucab commented Aug 30, 2020

lmachucab commented Aug 30, 2020

moisesbr-dw commented Jun 1, 2020 •

edited

Klap-in commented Aug 26, 2020 •

edited