Words of messages in some languages not found on search result

[luixxiul]

Steps to reproduce

Run the test below on timeline.spec.ts.

it("should highlight search result words of various languages", () => {
    // "Test" in Arabic, Hebrew, and Hindi
    const stringAr = "اِمْتِحَان‎";
    const stringHe = "מִבְחָן";
    const stringHi = "आज़माइश";

    cy.visit("/#/room/" + roomId);

    // Wait until configuration is finished
    cy.contains(
        ".mx_RoomView_body .mx_GenericEventListSummary .mx_GenericEventListSummary_summary",
        "created and configured the room.",
    ).should("exist");

    // Arabic
    cy.sendEvent(roomId, null, "m.room.message" as EventType, {
        msgtype: "m.text" as MsgType,
        body: stringAr,
    });
    // Hebrew
    cy.sendEvent(roomId, null, "m.room.message" as EventType, {
        msgtype: "m.text" as MsgType,
        body: stringHe,
    });
    // Hindi
    cy.sendEvent(roomId, null, "m.room.message" as EventType, {
        msgtype: "m.text" as MsgType,
        body: stringHi,
    });

    // Ensure the last message was sent
    cy.get(".mx_EventTile_last .mx_EventTile_receiptSent").should("be.visible");

    cy.get(".mx_RoomHeader_searchButton").click();

    // Check stringAr is highlighted
    cy.get(".mx_SearchBar_input input").clear().invoke("val", stringAr).trigger("input");
    cy.get(".mx_SearchBar_input input").type("{enter}");
    cy.get(".mx_EventTile:not(.mx_EventTile_contextual) .mx_EventTile_searchHighlight").should("exist");

    // Check stringHe is highlighted
    cy.get(".mx_SearchBar_input input").clear().invoke("val", stringHe).trigger("input");
    cy.get(".mx_SearchBar_input input").type("{enter}");
    cy.get(".mx_EventTile:not(.mx_EventTile_contextual) .mx_EventTile_searchHighlight").should("exist");

    // Check stringHi is highlighted
    cy.get(".mx_SearchBar_input input").clear().invoke("val", stringHi).trigger("input");
    cy.get(".mx_SearchBar_input input").type("{enter}");
    cy.get(".mx_EventTile:not(.mx_EventTile_contextual) .mx_EventTile_searchHighlight").should("exist");
});

Outcome

What did you expect?

The test should run successfully.

What happened instead?

It fails due to no results being found.

Please note that the similar test for other non-European languages like Chinese, Japanese, Korean, and Thai passes. Also the test for the Hebrew string without symbols (מבחן) also passes as well.

Operating system

Debian

Browser information

Electron (cypress)

URL for webapp

No response

Application version

develop branch

Homeserver

No response

Will you send logs?

No

[luixxiul]

In fact the issue is not really about highlighting words but finding them.

[justjanne]

This sounds like tokenization in search is broken?

[luixxiul]

Though I have not found yet how to reproduce consistently (and I am not quite sure if this is really related to this issue), it is almost impossible to search Japanese words in a room created and used actively since a half year.

[richvdh]

Assuming this is server-side search, it sounds more like a server-side issue than a client-side one.

matrix-org/synapse#901, possibly?