Bar Store

Bar Store

ib.bar_store.BarStore is a SQLite-backed historical bar cache. It tracks which date ranges have already been fetched from IB (coverage), so repeated requests for the same range never hit IB rate limits. When part of a requested range is missing (gap), only the gap is fetched — not the whole range.

---

Concepts

Series — a unique combination of (symbol, bar_size, what_to_show, use_rth). Each series has its own coverage table and bar rows.

Coverage — a set of non-overlapping [start, end] UTC intervals recorded after each successful fetch. Stored in the coverage table; automatically merged on every update.

Gap — a sub-interval of the requested range that has no coverage. get_bars calls the caller-supplied fetch_fn once per gap (further split into IB-safe chunks if the gap is wide).

BarRecord — the namedtuple returned by get_bars:

BarRecord(date, open, high, low, close, volume, wap, bar_count)

date is the original IB date string (e.g. "20260319 09:30:00 US/Eastern").

---

API Reference

BarStore(db_path)

Open (or create) a SQLite cache at db_path. Creates parent directories. Initialises the schema on first open.

from ib.bar_store import BarStore
store = BarStore("historical/gld_5min.db")

---

get_bars(...) → List[BarRecord]

bars = store.get_bars(
    symbol      = "GLD",
    bar_size    = "5 mins",
    what_to_show= "TRADES",
    use_rth     = True,
    start_dt    = datetime(2026, 1, 1, tzinfo=UTC),
    end_dt      = datetime(2026, 3, 1, tzinfo=UTC),
    fetch_fn    = my_fetch_fn,   # callable(start_dt, end_dt) → list of bar objects
    force       = False,         # True → bypass cache, re-fetch and overwrite
)

fetch_fn receives UTC-aware start_dt / end_dt datetimes and must return a list of objects with .date, .open, .high, .low, .close, .volume (and optionally .wap, .barCount). Returning an empty list is valid; coverage is still recorded for the chunk so it won't be re-requested.

force=True treats the full range as a single gap regardless of coverage. Useful after a corporate action or data correction.

Chunking: large gaps are automatically split into IB-safe chunks (e.g. 30 days for 5-min bars, 1 day for 1-sec bars). Each chunk is a separate fetch_fn call.

---

coverage_summary(symbol=None) → List[dict]

Returns one dict per series (optionally filtered by symbol):

for entry in store.coverage_summary(symbol="GLD"):
    print(entry["symbol"],    # "GLD"
          entry["bar_size"],  # "5 mins"
          entry["intervals"], # [("2026-01-01T00:00:00", "2026-03-01T00:00:00")]
          entry["total_bars"])

Key	Type	Description
symbol	str
bar_size	str
what_to_show	str
use_rth	bool
intervals	list of (start, end) str	ISO-8601 UTC strings
total_bars	int	Total rows in the bars table for this series

---

purge(symbol, bar_size, what_to_show, use_rth) → int

Delete all bars and coverage for a series. Returns the number of bar rows deleted. The next get_bars call will re-fetch from IB.

n = store.purge("GLD", "5 mins", "TRADES", True)
print(f"Deleted {n} bars")

---

duration_str(start_dt, end_dt) → str

Module-level helper. Converts a (start, end) datetime pair to an IB durationStr string ("N S|D|W|M|Y"). Use it inside fetch_fn:

from ib.bar_store import duration_str

def fetch_fn(start_dt, end_dt):
    return plugin.get_historical_data(
        contract=ContractBuilder.etf("GLD"),
        end_date_time=end_dt.strftime("%Y%m%d-%H:%M:%S"),
        duration_str=duration_str(start_dt, end_dt),
        bar_size_setting="5 mins",
        what_to_show="TRADES",
        use_rth=True,
    ) or []

---

Database Schema

Two tables, keyed on the series tuple (symbol, bar_size, what_to_show, use_rth):

bars (
    symbol, bar_size, what_to_show, use_rth,
    bar_dt_utc  TEXT PRIMARY KEY component,   -- ISO-8601 UTC
    bar_dt_orig TEXT,                          -- original IB date string
    open, high, low, close REAL,
    volume INTEGER, wap REAL, bar_count INTEGER
)

coverage (
    symbol, bar_size, what_to_show, use_rth,
    start_utc TEXT, end_utc TEXT,              -- ISO-8601 UTC
    fetched_at TEXT
)

WAL journal mode is enabled for concurrent read access.

---

CLI — ibctl.py historical

Fetch bars from IB through the running engine and optionally persist them to a BarStore DB. Useful for building test fixtures or pre-loading a cache.

historical fetch

./ibctl.py historical fetch SYMBOL [options]

Options:
  --bar-size "5 mins"      Bar width (default: "1 day")
  --duration "2 D"         How far back: N S|D|W|M|Y (default: "1 W")
  --end YYYYMMDD-HH:MM:SS  End datetime in UTC (default: now)
  --what TRADES            TRADES | MIDPOINT | BID | ASK (default: TRADES)
  --type etf               Contract type: etf (default) | stock | forex
  --no-rth                 Include extended-hours bars
  --db PATH                Save bars to a BarStore SQLite DB

Examples:

# Preview — fetch and print a table, no DB write
./ibctl.py historical fetch GLD
./ibctl.py historical fetch GLD --bar-size "5 mins" --duration "2 D"
./ibctl.py historical fetch EUR --type forex --what MIDPOINT --no-rth

# Fetch and persist
./ibctl.py historical fetch GLD --bar-size "5 mins" --duration "30 D" --db gld.db

Output:

[OK] Fetched 156 bars for GLD (5 mins, 2 D)

  Date                        Open      High       Low     Close      Volume
  ------------------------------------------------------------------------
  20260318 09:30:00 US/E    446.66    446.73    445.55    446.29     578,843
  ...
  20260319 15:55:00 US/E    426.18    427.07    426.07    426.43     533,196

[DB] Saved 156 bar(s) to gld.db  [GLD / 5 mins / TRADES]

historical coverage

./ibctl.py historical coverage --db PATH [--symbol SYMBOL]

Shows what is cached in a local BarStore DB. No engine connection needed.

Symbol   BarSize    What       RTH     Bars  Intervals
--------------------------------------------------------------------------------
GLD      5 mins     TRADES     Y        156  2026-03-18T13:30:00..2026-03-19T21:00:00

historical purge

./ibctl.py historical purge --db PATH --symbol SYMBOL \
    [--bar-size "5 mins"] [--what TRADES] [--no-rth]

Deletes a series from a local BarStore DB. No engine connection needed.

---

Using BarStore in a Plugin

See Plugin Manual §9.1 for the full in-plugin usage guide and API summary.

Quick pattern:

from ib.bar_store import BarStore, duration_str
from ib.contract_builder import ContractBuilder
from zoneinfo import ZoneInfo
from datetime import datetime, timedelta

UTC = ZoneInfo("UTC")

class MyPlugin(PluginBase):
    def start(self) -> bool:
        self._store = BarStore(self._base_path / "bars.db")
        return True

    def _get_bars(self, symbol, days=30):
        end_dt   = datetime.now(UTC)
        start_dt = end_dt - timedelta(days=days)

        def fetch(s, e):
            return self.get_historical_data(
                contract=ContractBuilder.etf(symbol),
                end_date_time=e.strftime("%Y%m%d-%H:%M:%S"),
                duration_str=duration_str(s, e),
                bar_size_setting="5 mins",
                what_to_show="TRADES",
                use_rth=True,
            ) or []

        return self._store.get_bars(
            symbol=symbol, bar_size="5 mins",
            what_to_show="TRADES", use_rth=True,
            start_dt=start_dt, end_dt=end_dt,
            fetch_fn=fetch,
        )

---

Paper Test

plugins/paper_tests/paper_test_bar_store/ verifies BarStore end-to-end against a live paper account. Run it with:

python run_paper_tests.py --bar-store

Test	What it verifies
cold_fetch_gld	Cold cache → IB fetched → bars returned + coverage set
cache_hit_gld	Same range second call → no IB fetch (call count stays 0)
gap_fill_gld	Cache middle day; wider request fills surrounding gaps without re-fetching cached portion
gap_in_middle	Cache outer wings; full-range request fetches only the middle hole
force_refetch_gld	force=True always calls IB even when fully cached
coverage_summary	coverage_summary() returns correct symbol, bar_size, interval, bar count
purge_and_refetch	purge() clears cache; next call re-fetches from IB
multi_symbol	GLD and UUP cached independently — no cross-contamination
ohlc_valid	All returned BarRecords have valid OHLC and positive volume