fix: make the Typesense search backend usable

[blarghmatey]

Description

Every forum search on the Typesense backend fails against a real Typesense server. There are two independent defects, both present since the backend shipped in #225, and either one alone is enough to break search.

1. per_page=1000 exceeds Typesense's hard cap of 250. build_search_parameters() passes per_page=FORUM_MAX_DEEP_SEARCH_COMMENT_COUNT (1000) on every search. Typesense caps per_page at 250 and rejects anything higher with HTTP 422, so this fires on every search, filtered or not.

The constant is shared with the other two backends, which is how it slipped through. Elasticsearch passes it as size (es.py:670) under index.max_result_window, and Meilisearch as limit (meilisearch.py:198) under maxTotalHits. Neither is set anywhere in this repo, so both take their product default — 10000 and 1000 respectively, read back from the e2e containers to check. Only Typesense caps low enough to break, and Meilisearch only just clears it.

Rather than quietly lower the deep-search budget for this backend alone, get_thread_ids() now walks pages of 250 until the budget is covered or a short page says the results are exhausted. Searches returning under 250 hits — nearly all of them — still cost exactly one request.

2. The topic filter names a field the collection does not have. collection_schema() declares commentable_id; build_search_parameters() filtered on commentable_ids, so any search scoped to discussion topics failed with HTTP 400 Could not find a filter field named 'commentable_ids' in the schema. Corrected to the singular name the collection actually declares. No reindex needed — document_from_thread() always wrote the field under the right name, only the filter was wrong.

Testing

Measured against Typesense 30.2, replicating build_search_parameters() against a collection built from collection_schema() and holding 600 threads:

search parameters	result
per_page=1000 + commentable_ids (as shipped)	400 Could not find a filter field
per_page=250 + commentable_ids (bug 2 alone)	400 Could not find a filter field
per_page=1000 + commentable_id (bug 1 alone)	422 Only upto 250 hits can be fetched per page
per_page=250 + commentable_id, pages 1–3	200, found=600, 250 + 250 + 100 hits
get_thread_ids() after this change	600 of 600 threads returned

Neither defect was reachable from the existing tests, which mock the Typesense client and so cannot observe a rejection from the server — the unit test asserting commentable_ids and per_page=1000 passed happily against both bugs. That gap is the reason this shipped, so this PR closes it: tests/e2e/ gains a Typesense service in its compose file and a test_search_typesense.py covering both defects against a real server. Three of the four new tests fail on master with exactly the errors above (the fourth only builds and validates a collection, so it passes either way):

tests/e2e/test_search_typesense.py::test_insert_document
E   typesense.exceptions.ObjectUnprocessable: [Errno 422] Only upto 250 hits can be fetched per page.
tests/e2e/test_search_typesense.py::test_search_filtered_by_commentable_id
E   typesense.exceptions.RequestMalformed: [Errno 400] Could not find a filter field named `commentable_ids` in the schema.
tests/e2e/test_search_typesense.py::test_deep_search_past_the_per_page_limit
E   typesense.exceptions.RequestMalformed: [Errno 400] Could not find a filter field named `commentable_ids` in the schema.
3 failed, 1 passed

(That run used master's typesense.py with only TYPESENSE_MAX_PER_PAGE = 250 appended, since the new tests reference that constant. It adds a name, not behaviour.)

test_insert_document is worth a look: it is an ordinary unfiltered search, and it fails too. Forum search on this backend is non-functional as shipped, not merely degraded for topic-scoped queries.

With the fix: pytest tests 200 passed, pytest tests/e2e 42 passed, and mypy, pylint, pycodestyle and pydocstyle are clean over src/forum tests test_utils manage.py.

TYPESENSE_URLS in the test settings pointed at 0.0.0.0:8108, where nothing runs. It now points at the new e2e container, mapped to 5108 to match the port-shifting convention Elasticsearch (5200) and Meilisearch (5700) already follow there.

Scope

Out of scope, unchanged, and already noted in the backend's own comments: get_suggested_text() still returns None, so there is no "did you mean" spelling correction here, which the Elasticsearch backend does provide via phrase suggesters (es.py:545); group_ids and sort_criteria also remain unsupported. Anyone moving an environment onto this backend should weigh the loss of spelling correction separately.

I could not find an existing issue or PR covering either defect: no issue matches typesense, per_page or commentable_ids in any state, and the only PRs to have touched search/typesense.py are #225 itself plus #268 and #283, which dropped Python 3.11 and moved the repo to uv and a src layout. Neither buggy line has changed since #225 introduced it.

Merge checklist:
Check off if complete or not applicable:

• Version bumped — N/A, semantic-release derives it from the fix: commit
• Changelog record added — N/A, same
• Documentation updated (not only docstrings)
• Fixup commits are squashed away
• Unit tests added/updated
• Manual testing instructions provided
• Noted any: Concerns, dependencies, migration issues, deadlines, tickets

[openedx-webhooks]

Thanks for the pull request, @blarghmatey!

This repository is currently maintained by @openedx/wg-maintainers-forums.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

• If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
  • This process (including the steps you'll need to take) is documented here.
• If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

• Dependencies

  This PR must be merged before / after / at the same time as ...

• Blockers

  This PR is waiting for OEP-1234 to be accepted.

• Timeline information

  This PR must be merged by XX date because ...

• Partner information

  This is for a course on edx.org.

• Supporting documentation
• Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details

---

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

• Overview of Review Process for Community Contributions
• Pull Request Status Guide
• Making changes to your pull request

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

• The size and impact of the changes that it introduces
• The need for product review
• Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

[blarghmatey]

@openedx/wg-maintainers-forums — this is ready for engineering review whenever someone has time. All checks are green, including the new e2e job.

Short version of why it might be worth prioritising: the Typesense search backend is currently non-functional for every search, not just some. Two independent defects, both present since the backend shipped in #225:

• per_page is set to FORUM_MAX_DEEP_SEARCH_COMMENT_COUNT (1000), and Typesense rejects anything above 250 with HTTP 422. This fires on every search, filtered or not.
• The topic filter uses commentable_ids, but the collection schema declares commentable_id, so any topic-scoped search fails with HTTP 400.

Neither was reachable from the existing tests, which mock the Typesense client and so cannot observe a rejection from the server — the unit test asserting the broken parameters passed happily. That is why the PR adds a Typesense service to the e2e compose file and covers both defects against a real server; three of the four new tests fail on master with exactly those errors.

The change is confined to search/typesense.py plus its tests, and touches neither the Elasticsearch nor the Meilisearch backend, so the blast radius is limited to operators who have explicitly opted into Typesense for forum search.

Happy to rebase or split it if that helps review. No rush on our end beyond wanting it upstream rather than carried as a patch.