refactor(llmrails): extract generation workflow (4/9)

[Pouyanpi]

Description

Moves request normalization, request context, response shaping, tracing, and the standard generation workflow into generation helpers.

Why

generate_async becomes orchestration around an owned generation boundary while preserving the public API.

What Changed

• Extracts generation request normalization and public-state validation.
• Extracts request context lifecycle and bot-message assembly.
• Extracts response construction, tracing, and standard generation execution.
• Preserves current develop assistant tool-call handling.

Review Guidance

Use the LLMRails decomposition review matrix to validate generation, options, response shaping, tool-call, tracing, and retrieval scenarios against the recorded #2122 baseline.

Focus on validation order, context cleanup, metadata capture, cache behavior, response fields, and delegation equivalence. Tracing ownership is intentionally consolidated under generation in part 8.

This nine-part stack turns LLMRails into a public compatibility shell while moving owned behavior into focused modules. Review each PR only against its parent. Parts 2-6 intentionally stage behavior-preserving extraction; part 7 consolidates shared type contracts and logger naming; part 8 establishes the final package ownership; part 9 contains the explicit semantic follow-ups.

For parts 2-6, please distinguish stack-local correctness or compatibility problems from naming and placement concerns already resolved in parts 7-8. If an intermediate name or location is correct and mergeable at that point, prefer reviewing its final form in the resolving PR instead of requesting duplicate churn. Public API changes, behavior changes, import cycles, or an independently broken intermediate branch remain in scope wherever they appear. Much of the apparent added LOC is moved code plus focused tests; the corresponding implementation is removed or delegated from llmrails.py in the same PR.

Stack Position

Part 4 of 9.

• Previous: #2125
• Next: #2127

Order	PR	Branch	Base
1	#2123	restack/llmrails-01-public-contract-tests	develop
2	#2124	restack/llmrails-02-startup	restack/llmrails-01-public-contract-tests
3	#2125	restack/llmrails-03-runtime-conversation	restack/llmrails-02-startup
4	#2126	restack/llmrails-04-generation	restack/llmrails-03-runtime-conversation
5	#2127	restack/llmrails-05-streaming	restack/llmrails-04-generation
6	#2128	restack/llmrails-06-checks-final	restack/llmrails-05-streaming
7	#2129	restack/llmrails-07-consistency	restack/llmrails-06-checks-final
8	#2130	restack/llmrails-08-boundary-ownership	restack/llmrails-07-consistency
9	#2131	restack/llmrails-09-followups	restack/llmrails-08-boundary-ownership

Verification

make test
poetry run pre-commit run --all-files

Rebase checkpoints: stack 2: 5202 passed, 178 skipped; stack 4: 5263 passed, 178 skipped; stack 5: 5284 passed, 178 skipped; final stack: 5320 passed, 178 skipped. Final pre-commit: all hooks passed. Base: develop at 3fc828b7f.

AI Assistance

• No AI tools were used.
• AI tools were used; a human reviewed and can explain every change (tool: Codex).

Checklist

• I've read the CONTRIBUTING guidelines.
• This PR links to a triaged issue assigned to me.
• My PR title follows the project commit convention.
• I've updated the documentation if applicable.
• I've added tests if applicable.
• I've noted any verification beyond CI and any checks I couldn't run.
• I did not update generated changelog files manually.
• I addressed all CodeRabbit, Greptile, and other review comments, or replied with why no change is needed.
• @mentions of the person or team responsible for reviewing proposed changes.

[codecov]

Codecov Report

❌ Patch coverage is 99.25187% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...rdrails/rails/llm/generation/generation_tracing.py	94.87%	2 Missing ⚠️
...drails/rails/llm/generation/generation_response.py	98.59%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!