search_dates() raises IndexError for the zh-Hans, zh-Hant and yue locales

[serhii73]

Problem

search_dates() raises IndexError: list index out of range for the zh-Hans, zh-Hant and yue locales. It is reachable through the plainest possible call, with no languages= argument at all:

>>> from dateparser.search import search_dates
>>> search_dates("聽日")          # Cantonese for "tomorrow"; autodetection selects yue
IndexError: list index out of range
>>> search_dates("舊年")          # "last year"
IndexError: list index out of range

It also fires whenever one of the three locales is requested explicitly, including in a mixed language list, and including for plain ASCII input:

>>> search_dates("今天", languages=["zh-Hans"])                 # IndexError
>>> search_dates("2020年3月4日", languages=["zh-Hant"])          # IndexError
>>> search_dates("00Z", languages=["yue"])                      # IndexError
>>> search_dates("2020年3月4日", languages=["zh-Hans", "en"])    # IndexError
>>> search_dates("3 hours ago at 5pm", languages=["zh-Hans"])   # IndexError

zh, ja and every other locale are unaffected, and dateparser.parse() does not crash (for these locales it returns None instead — that half is #1277).

Traceback:

dateparser/search/__init__.py:70  -> search/search.py:357
                                  -> search/search.py:184
                                  -> search/search.py:45
                                  -> languages/locale.py:310
      [original_tokens[i], original_tokens[i + 1]],
IndexError: list index out of range

Root cause

Two things combine in Locale.translate_search() (dateparser/languages/locale.py):

1. locale.py:277 — word_joint_unsupported_languages = ["zh", "ja"]. Exactly zh and ja carry a simplifications block that rewrites 年/月/日 into separators before this point; zh-Hans, zh-Hant and yue do not, so they reach the word-joining branch at locale.py:302-312 with raw CJK tokens, which the other two never do.
2. locale.py:291 computes next_word defensively — simplified_tokens[i + 1] if i < last_token_index else "" — but the branch it guards reads original_tokens[i + 1] at locale.py:310 with no equivalent bounds check. On the final token, current_and_next_joined can still match a dictionary entry (the joined form equals the bare word, since next_word is ""), the branch is taken, and the index overflows.

Introduced in c5f0fa4 (#953, 2021-08-09); git show c5f0fa4^:dateparser/languages/locale.py contains no i + 1 index at all.

Proposal

Add the missing bounds check to the condition at locale.py:302:

                elif (
                    i < last_token_index
                    and current_and_next_joined in dictionary
                    and word not in dashes
                    and self.shortname not in word_joint_unsupported_languages
                ):

I verified this locally:

• every repro above stops raising;
• the full suite is unchanged — 24186 passed, 18 skipped, 1 xfailed, identical to baseline;
• search_dates("聽日") and search_dates("舊年") now return correct results rather than merely not crashing.

Scope and considerations

• Please do not fix this by adding the three locales to word_joint_unsupported_languages. I tried that: it stops the crash, but search_dates("2020年3月4日", languages=["zh-Hans"]) then returns datetime(6, 5, 27) — silent garbage — because translate_search yields "2020yearmarch4day". That trades a loud crash for a wrong date.
• Related but distinct: Aren't "zh" (Chinese) and "zh-Hans" (Chinese, traditional script) the same language? #1277 (zh-Hans vs zh locale data) and Issue parsing ZH-Hant locale #875 (zh-Hant). Those are data gaps that make parse() return None; this is a code defect that survives them. I applied Aren't "zh" (Chinese) and "zh-Hans" (Chinese, traditional script) the same language? #1277's proposed change in-memory and search_dates() still raises for 今天, 昨天, 上个月 and 00Z. Sensible sequencing is the bounds guard first (crash → None, matching parse()), the data fix second (None → correct date).
• After the guard lands, 2020年3月4日 under zh-Hans returns None rather than a date. That is Aren't "zh" (Chinese) and "zh-Hans" (Chinese, traditional script) the same language? #1277's territory, not a regression from this change.

Acceptance criteria

• None of the calls listed above raises IndexError.
• Regression tests cover zh-Hans, zh-Hant and yue in tests/test_search.py, which currently contains zero references to any of the three (grep -c 'zh-Han\|yue' tests/test_search.py → 0) — which is how this survived four and a half years.