OLS-3799: Fix permanent Solr connection loss when RHOKP starts slowly

[blublinsky]

Description

Summary
When RHOKP takes longer than the retry window to start, the app-server permanently loses product documentation RAG with no recovery short of a pod restart. This PR fixes the root cause and simplifies the startup sequence.

Root Cause
solr_hybrid_search was a @cached_property — once it resolved to None (Solr unreachable), it stayed None forever
Index loading ran on a background thread, racing with request handling — a request could arrive before initialization completed

Changes

Removed background threading.Thread for index loading — load_index() now runs synchronously before start_uvicorn(). Eliminates race conditions between initialization and request handling. The readiness probe budget (8 min) accommodates the startup delay.
ols/utils/config.py

Changed solr_hybrid_search from @cached_property to @Property with manual caching. On failure, returns None without caching — the next request re-attempts initialization. After 3 failed attempts, gives up permanently with an error log. On success, caches the result for the lifetime of the process.
ols/src/rag_index/solr_support.py

Increased _SOLR_STARTUP_RETRIES from 24 to 36 (120s → 180s) to better align with RHOKP's startup probe budget (6 min).
tests/unit/query_helpers/test_docs_summarizer.py

Updated test to use the new caching attributes instead of dict manipulation.

Result
RHOKP starting after the app-server is tolerated without a pod restart
No race between background thread and request handling
Bounded retry: 3 attempts × 180s = 9 minutes max before permanent give-up with clear error log

Type of change

• [ x] Refactor
• New feature
• Bug fix
• CVE fix
• Optimization
• Documentation Update
• Configuration Update
• Bump-up dependent library
• Bump-up library or tool used for development (does not change the final image)
• CI configuration change
• Konflux configuration change

Related Tickets & Documents

• Related Issue #
  https://redhat.atlassian.net/browse/OLS-3799
• Closes #
  https://redhat.atlassian.net/browse/OLS-3799

Checklist before requesting a review

• I have performed a self-review of my code.
• PR has passed all pre-merge test jobs.
• If it is a core feature, I have added thorough tests.

Testing

• Please provide detailed steps to perform tests related to this code change.
• How were the fix/results from this change verified? Please provide relevant screenshots or results.

[openshift-ci-robot]

@blublinsky: This pull request references OLS-3799 which is a valid jira issue.

Details

In response to this:

Description

Summary
When RHOKP takes longer than the retry window to start, the app-server permanently loses product documentation RAG with no recovery short of a pod restart. This PR fixes the root cause and simplifies the startup sequence.

Root Cause
solr_hybrid_search was a @cached_property — once it resolved to None (Solr unreachable), it stayed None forever
Index loading ran on a background thread, racing with request handling — a request could arrive before initialization completed

Changes

Removed background threading.Thread for index loading — load_index() now runs synchronously before start_uvicorn(). Eliminates race conditions between initialization and request handling. The readiness probe budget (8 min) accommodates the startup delay.
ols/utils/config.py

Changed solr_hybrid_search from @cached_property to @Property with manual caching. On failure, returns None without caching — the next request re-attempts initialization. After 3 failed attempts, gives up permanently with an error log. On success, caches the result for the lifetime of the process.
ols/src/rag_index/solr_support.py

Increased _SOLR_STARTUP_RETRIES from 24 to 36 (120s → 180s) to better align with RHOKP's startup probe budget (6 min).
tests/unit/query_helpers/test_docs_summarizer.py

Updated test to use the new caching attributes instead of dict manipulation.

Result
RHOKP starting after the app-server is tolerated without a pod restart
No race between background thread and request handling
Bounded retry: 3 attempts × 180s = 9 minutes max before permanent give-up with clear error log

Type of change

• [ x] Refactor
• New feature
• Bug fix
• CVE fix
• Optimization
• Documentation Update
• Configuration Update
• Bump-up dependent library
• Bump-up library or tool used for development (does not change the final image)
• CI configuration change
• Konflux configuration change

Related Tickets & Documents

• Related Issue #
  https://redhat.atlassian.net/browse/OLS-3799
• Closes #
  https://redhat.atlassian.net/browse/OLS-3799

Checklist before requesting a review

• I have performed a self-review of my code.
• PR has passed all pre-merge test jobs.
• If it is a core feature, I have added thorough tests.

Testing

• Please provide detailed steps to perform tests related to this code change.
• How were the fix/results from this change verified? Please provide relevant screenshots or results.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

Warning

Review limit reached

@blublinsky, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 3 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 21218309-16de-4d59-b718-eb921d4f3156

📥 Commits

Reviewing files that changed from the base of the PR and between c733eaa and ad2a7cd.

📒 Files selected for processing (4)

• ols/src/rag_index/solr_support.py
• ols/utils/config.py
• runner.py
• tests/unit/query_helpers/test_docs_summarizer.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[raptorsun]

/lgtm
/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: raptorsun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [raptorsun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[raptorsun]

concurrent request can burn out the retry count quickly, but it should rarely happen

[openshift-ci]

@blublinsky: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.