Issue 102 Search Found Nothing For Hyphenated Values

Search found nothing for values containing a hyphen (issue #102)

A customer number like 05145-8841815, sitting in a spreadsheet attached to a ticket, could not be found by typing it. Typing the very same value with a space instead of the hyphen — 05145 8841815 — found it immediately.

Reported in issue #102 by dschipfel, with a table of seven searches and their results that made the diagnosis possible.

Fixed in b45f8a8d, released as update #1200.

---

1. What you saw

The reporter listed exactly which searches worked and which did not:

Typed	Result
05145-8841815	no results
051458841815	no results
41815	no results
05145,884	found
05145.884	found
05145 8841815	found
05145 884	found

Read that table again and the shape of it is odd: a comma works, a full stop works, a space works — and a hyphen does not. Those are all punctuation. Whatever was wrong could not simply be "punctuation confuses it".

---

2. The mechanism

Two things split text into words: the index, and the search box. They disagreed.

MySQL splits on punctuation. The text 05145-8841815 is stored in the index as two separate words:

05145        8841815

There is no entry for 05145-8841815, and none for 051458841815 either.

FreeITSM deleted the punctuation instead of splitting on it:

// includes/search/search.php — before
$clean = preg_replace('~[+\-><()\~*@"]+~u', '', $word);

So 05145-8841815 became the single word 051458841815, and that is what was looked for:

+051458841815*

A word that exists nowhere in the index. The search was not confused — it did precisely what it was told, and correctly reported that nothing matched.

Why a comma worked and a hyphen did not

The strip list is + - > < ( ) ~ * @ " — the MySQL boolean operators. A comma is not on it. Nor is a full stop. They survived into the query and MySQL, which treats them as separators, split there itself. So the comma and full stop accidentally did the right thing, and the hyphen — being on the list — was destroyed.

That is the whole reason the table looks inconsistent. The behaviour depended on which punctuation mark happened to be an operator.

The file already contained the right answer, nine lines above

Quoted phrases are handled by a separate branch, which does the same substitution with one difference:

// the quoted-phrase branch — replaces with a SPACE
$p = trim(preg_replace('~[+\-><()\~*@"]+~u', ' ', $p));

// the word branch — replaced with NOTHING
$clean = preg_replace('~[+\-><()\~*@"]+~u', '', $word);

' ' against ''. One character, nine lines apart. Searching for "05145-8841815" with quotes worked correctly the entire time.

---

3. The fix

Split on anything that is not a word character, rather than deleting it:

// after
$bits = preg_split('~[^\p{L}\p{N}_]+~u', (string)$word, -1, PREG_SPLIT_NO_EMPTY);

Letters, digits and underscore are word characters, mirroring what InnoDB's own parser treats as one. Everything else — hyphen, dot, comma, slash, colon, apostrophe, and the boolean operators, which are ours to add rather than the user's to inject — is a separator.

before: 05145-8841815  ->  +051458841815*      no results
after:  05145-8841815  ->  +05145* +8841815*   found

A hyphenated value now reads exactly as if you had typed a space, which is what the reporter's own workaround already relied on. Every separator behaves the same way, so the inconsistency in that table is gone.

The leading minus is still the exclusion operator, and now applies to every part of the word it is attached to: -pre-flight becomes -pre -flight.

Quoting still demands adjacency, and is now the documented way to be precise:

"05145-8841815"  ->  +"05145 8841815"

---

4. Two of the seven are not fixed

Worth stating plainly, because both are limits of the search engine rather than decisions.

051458841815 — punctuation omitted entirely. The stored text contains a hyphen. Nothing in the index corresponds to the digits run together, so there is nothing to match. This could be made to work by also indexing a punctuation-stripped copy of every document, at the cost of a larger index and a full reindex of everything already stored. Not done.

41815 — the tail of a single word. MySQL full-text search matches the start of a word (884* finds 8841815) and never the end. There is no wildcard for it.

But the limit is narrower than the report supposed. A "latter part" that is a whole word works perfectly well: searching 789 finds GHI-789, because 789 is indexed as a word in its own right. It only fails when the fragment sits inside one word, as 41815 does inside 8841815.

The practical workaround for a long number is to type the part you know from its beginning: 05145 884 finds it.

---

5. 📁 The files involved

File	What changed
includes/search/search.php	searchParseQuery() splits on non-word characters instead of deleting them.
tests/search/run.php	Eight new cases.

🗄️ Already correct

The indexer	It never had this fault. It stores the text as it is; MySQL does the splitting.
The quoted-phrase branch	Always substituted a space. "05145-8841815" worked throughout.

---

6. How it was verified

Reproduced first. A fixture containing the reporter's exact value, run against all seven of their search terms, produced their table line for line — including pre-flight, which they noted as the same fault in ordinary words.

End to end on a real emailed attachment. An Excel file was emailed into a test installation, arriving as TICKET-000108, containing ABC-123 DEF-456 GHI-789 XXX-YYY-ZZZ. Searched through the real query path with real scoping:

Search	Before	After
ABC-123	+ABC123* — nothing	+ABC* +123* — found
DEF-456	+DEF456* — nothing	+DEF* +456* — found
XXX-YYY-ZZZ	+XXXYYYZZZ* — nothing	+XXX* +YYY* +ZZZ* — found

That last one matters: a value with two hyphens splits into three parts correctly.

The test suite. tests/search/run.php went from 44 passing to 52, the eight new cases covering the split itself, every separator behaving alike, the leading minus across a hyphenated word, operators embedded mid-word, and a fragment below the index minimum being dropped rather than required.

That last one guards a real trap the suite already knew about: requiring a word shorter than the index minimum makes the entire query match nothing, so B4-05145 must search for 05145 alone rather than demanding a B4 that was never indexed.

---

7. What this means for you

Type a reference with hyphens in it and it will be found — in ticket text, notes, and the contents of attached documents.

If you want to be precise, put quotes round it: "05145-8841815" requires the parts to appear together, in that order, rather than merely somewhere in the same document.

If you only know part of a long number, type it from the start (05145 884) rather than the end.

---

Related pages

• Searching inside tickets
• Bugs resolved