[Interactive Brokers] v1 historical backfill can silently return a partial result — is the v2 Rust path the intended remedy?

[M-at-ti-a]

Context

I'm using HistoricInteractiveBrokersClient.request_bars() on v1.230.0 to build daily-bar research datasets. I hit a case where the returned series was shorter than the requested window with no error raised, and traced it to the chunking loop.

Before opening an issue I'd like to check whether this is considered v1-legacy-only, since #4484 and #4485 were recently closed on that basis.

What I observe (v1.230.0)

nautilus_trader/adapters/interactive_brokers/data.py, get_historical_bars_chunked:

for segment_end_date_time, segment_duration in segments:
    bars = await self._client.get_historical_bars(...)

    if bars:
        self._log.info(f"{bar_type.instrument_id}: Number of bars retrieved in batch: {len(bars)}")
        data.extend(bars)
        self._log.info(f"Total number of bars in data: {len(data)}")
    else:
        self._log.info(f"{bar_type.instrument_id}: No bars retrieved for: {bar_type}")

return sorted(data, key=lambda x: x.ts_init)

and client/client.py, _await_request:

except TimeoutError as e:
    ...
    return default_value
except ConnectionError as e:
    ...
    return default_value

So when a segment times out or hits a connection error, the failure becomes an empty list, the loop logs at INFO — the same line a genuinely empty window produces — and continues to the next segment. request_bars then does if bars: data.extend(bars) and returns the sorted survivors.

The result is well-formed, sorted, and short. Nothing in the return value or the exception behaviour distinguishes "IB has no data for that window" from "that segment failed".

Why this one seems worth raising

HistoricInteractiveBrokersClient is the research/backfill utility, so its output typically gets persisted and then used as the input to backtests. A silently short dataset doesn't produce a visible failure — it produces a plausible-looking backtest over less history than intended. If the shortfall lands at either edge of the requested window, it is also not detectable from the returned data alone, since the delivered series is internally contiguous.

v2 appears to already handle this

crates/adapters/interactive_brokers/src/historical/client.rs on develop:

let historical_data = tokio::time::timeout(
    std::time::Duration::from_secs(timeout),
    self.ib_client
        .historical_data(&contract, ib_bar_size)
        .ending(chrono_to_ib_datetime(&segment_end))
        .duration(segment_duration)
        .what_to_show(ib_what_to_show)
        .trading_hours(trading_hours)
        .fetch(),
)
.await
.context(format!(
    "Historical data request timed out after {} seconds",
    timeout
))??;

The ?? propagates rather than continuing, so a failed segment aborts the request with an error instead of yielding a partial result.

Question

Given that #4484 and #4485 were closed as v1-legacy with v1 retirement imminent:

1. Is the v2 Rust path the intended remedy here too — i.e. should v1 users treat "verify the delivered range against the range you requested, yourself" as the supported approach until they migrate?
2. Or would a minimal v1 guard be welcome — raising on a failed segment, or returning some partial-result indicator? Happy to open an issue and/or a PR if so.

I've raised this as a discussion rather than an issue because I suspect the answer is (1), but the silent-shortfall behaviour seemed worth having on the record somewhere searchable. Thanks for all the work on this project.

[M-at-ti-a]

Correction to my own post — two things above are imprecise, and one of them is simply wrong. Sorry for the noise.

1. The log levels are wrong as I stated them. I wrote that the failure "is logged at INFO — the same line a genuinely empty window produces". That is not right. In client/client.py::_await_request, the underlying failure is logged at WARNING for TimeoutError and ERROR for ConnectionError (v1.230.0, _await_request; identical in 1.228.0):

except TimeoutError as e:
    self._log.debug(msg) if suppress_timeout_warning else self._log.warning(msg)
    ...
    return default_value
except ConnectionError as e:
    self._log.error(f"Connection error during {request}; ending request")
    ...
    return default_value

Only the downstream summary in data.py — "No bars retrieved for: {bar_type}" — is INFO, and that line is shared between a swallowed failure and a genuinely empty window. So the failure is distinguishable in the logs; what is not distinguishable is the return value. My argument only ever needed the second claim, and I overstated it into the first.

2. The multi-segment path needs start_date_time, not duration. _calculate_duration_segments short-circuits:

if duration:
    return [(end_date, duration)]

So a caller passing duration= gets exactly one segment, and a failure yields a fully-empty result rather than a short one — much easier to detect. The silent-partial only arises on the start_date_time= path, where the window is decomposed into year/day/second segments and the loop can lose one in the middle. Worth stating precisely, since it narrows who is actually affected.

What I think still stands: with start_date_time, a mid-window segment failure yields a well-formed, sorted, short list[Bar] with no exception and no partial-result indicator, so a caller cannot tell programmatically that its dataset is incomplete. The question in the original post is unchanged — is the v2 Rust path the intended remedy, or would a v1 guard be welcome?

Apologies for not checking the log levels before posting.