feat: setup per tool timeouts in parallel executor

[hwixley]

Context

So each tool in parallel executor has it's own max timeout to catch any throttling tools.

What

• Sets up generic 30s per-tool timeout in run_tools_parallel as a new func param
• Adds a unique run_id UUID to run_tools_parallel logs so that tool logs in CloudWatch indicate which parallel tool run they originate from

Have you written unit tests?

• Yes
• No (add why you have not)

Are there any specific instructions on how to test this change?

• Yes (if so provide more detail)
• No

1. Do chat which requires tool execution, ie. "search the web for..."
2. Validate tools execute and return results
3. Validate logs display correctly

Relevant links

[nora-errouhly]

looked at the code and the description of this PR above. The code doesn't actually implement individual timeouts that would catch throttled tools before they complete. A truly throttled tool would simply not complete within the overall timeout and would be skipped entirely. Basically the future.as_completed would only return completed tool executions. so if a tool is throttled, the tool won't be complete as the future.as_completed won't return this tool since the timeout was exceeded. Adding per-tool-timeout ensure that any already completed tool execution return the result within the per-tool-timeout window. in rare cases, the tool might be complete but it takes more time to retrieve results (for example large data) in which case the per-tool-timeout ensure the code does not wait for the results indefinitely.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 88.88889% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.06%. Comparing base (bd14f98) to head (9fc24ac).

Files with missing lines	Patch %	Lines
redbox/redbox/graph/nodes/sends.py	88.88%	8 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##              dev     #656   +/-   ##
=======================================
  Coverage   81.05%   81.06%           
=======================================
  Files         183      183           
  Lines        7352     7387   +35     
=======================================
+ Hits         5959     5988   +29     
- Misses       1393     1399    +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[nora-errouhly]

for sanity check, result from tests attached where two tools are called

[nora-errouhly]

Update: the per-tool-timeout that was implemented was related to timeout for results retrieval from tool, not the tool execution itself. I adjusted the code to enforce timeout on the tool execution, also keeping another timeout for results retrieval (useful if retrieved data is large)

[saisakul]

please add test case when 1 tool completes the request and 1 tool timeout.

[nora-errouhly]

added

[nora-errouhly]

increased per-tool-timeout to 60sec in case 30sec might be too short.
Ready for review

[hwixley]

LGTM 🚀

[nora-errouhly]

LGTM 🚀

@saisakul are you happy to approve? neither Harry or myself can approve this as we contributed to this PR