My consolidation always fails at duration 8.1m

[KonradHEXANT]

Are there any settings that will allow a longer timeout?

[Arthur031221]

There is a configurable consolidation timeout: HINDSIGHT_API_CONSOLIDATION_WALL_TIMEOUT, in seconds. It limits time without progress during a consolidation task; each completed batch restarts the clock. The current default is 7,200 seconds (two hours), so a failure at 8.1 minutes by itself does not indicate that the default wall timeout fired.

I checked the current configuration source:

curl -fsSL https://raw.githubusercontent.com/vectorize-io/hindsight/0cc0f39fc2a6cd889f15353ab3ebc0b3bb383cc3/hindsight-api-slim/hindsight_api/config.py | grep -nE 'ENV_CONSOLIDATION_WALL_TIMEOUT =|DEFAULT_CONSOLIDATION_WALL_TIMEOUT =|ENV_CONSOLIDATION_LLM_TIMEOUT ='
438:ENV_CONSOLIDATION_LLM_TIMEOUT = "HINDSIGHT_API_CONSOLIDATION_LLM_TIMEOUT"
999:ENV_CONSOLIDATION_WALL_TIMEOUT = "HINDSIGHT_API_CONSOLIDATION_WALL_TIMEOUT"
1893:DEFAULT_CONSOLIDATION_WALL_TIMEOUT = 7200  # seconds (2 hours)

Check the failed operation's error or worker log before changing the limit. If it says the task made no progress for a configured number of seconds and names HINDSIGHT_API_CONSOLIDATION_WALL_TIMEOUT, raise that setting on the worker and restart it. If the error names an LLM request timeout, the separate HINDSIGHT_API_CONSOLIDATION_LLM_TIMEOUT setting is relevant. The screenshot shows duration and failure state, but not which timeout or error caused the failure.

[KonradHEXANT]

But there are no error details, just "timeout error"

[Arthur031221]

The screenshot narrows this down. The failing span uses the LiteLLM provider, and its error is TimeoutError:. In the current Hindsight source, the default LLM request timeout is 120 seconds with three retries. The LiteLLM path waits 1, 2, and 4 seconds between those retries. Four timed-out requests plus those waits take 487 seconds, or 8.12 minutes, which closely matches the 8.1 minutes shown. The blank text after TimeoutError: does not give a more specific cause.

Please check the worker logs for this run for LiteLLM call exceeded timeout= and LiteLLM call timed out after. Those lines report the timeout and attempt count actually used. If they confirm a 120-second request timeout and that backend normally needs longer, set HINDSIGHT_API_CONSOLIDATION_LLM_TIMEOUT to a suitable value, such as 300 seconds, on the worker and restart it. That setting falls back to HINDSIGHT_API_LLM_TIMEOUT; HINDSIGHT_API_CONSOLIDATION_WALL_TIMEOUT is a separate limit and will not extend an individual LLM request. If the log messages differ, please share the relevant worker log lines and your Hindsight version, with any credentials removed.

[KonradHEXANT]

Thanks :) That works :)