as_pool_build_search_tokens strips off last character

[robert-ancell]

The following program:

   g_autoptr(AsPool) pool = as_pool_new ();
   g_auto(GStrv) tokens = as_pool_build_search_tokens (pool, "evince");
   for (guint i = 0; tokens[i] != NULL; i++) {
      g_printerr ("%s\n", tokens[i]);
   }

Prints out evinc, which seems to indicate the last character is being dropped from search terms.

[robert-ancell]

FYI @pwithnall - I noticed this in gnome-software and followed it to appstream.

[pwithnall]

(Note that @robert-ancell might not be available for the next week to follow up on this)

[pwithnall]

This is likely the result of stemming (as-stemmer.c)

[ximion]

I agree, and since AppStream caches stemmed tokens, this would be expected behaviour... Unless of course stemming is inconsistent, which it shouldn't be. Philip Withnall ***@***.***> schrieb am Fr., 1. Okt. 2021, 13:06:

…

This is likely the result of stemming (as-stemmer.c) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#361 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADF5PAYMCOLGCJ3GJFVL2DUEWI4DANCNFSM5FDZBKJQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[pwithnall]

@robert-ancell did you notice this as the result of a misbehaviour in gnome-software, or just coincidentally?

[robert-ancell]

I noticed this specifically when debugging searches in the snap plugin. When you search for evince only "evinc" is passed to the plugin. I thought this was initially the last keypress not being detected in the search entry, but then found it was this function that was causing the issue.

The search tokens are built in gs-plugin-loader.c, if they're supposed to drop information should that be done at a later stage?

[ximion]

@robert-ancell I think that really depends on the description of GS' plugin API: Are the plugins supposed to get preprocessed (and thereby stemmed) search tokens, or are they supposed to get the raw input submitted by the user?
I think an argument could be made for both, the former is more efficient, the latter is more flexible as some plugins may want their own stemming.
In any case, this isn't an issue in AppStream, that pretty much works as intended as we do want stemmed search terms.