Bar Store

Bar Store

ib.bar_store.BarStore is a SQLite-backed historical bar cache. It tracks which date ranges have already been fetched from IB (coverage), so repeated requests for the same range never hit IB rate limits. When part of a requested range is missing (gap), only the gap is fetched — not the whole range.

---

Concepts

Series — a unique combination of (symbol, bar_size, what_to_show, use_rth). Each series has its own coverage table and bar rows.

Coverage — a set of non-overlapping [start, end] UTC intervals recorded after each successful fetch. Stored in the coverage table; automatically merged on every update.

Gap — a sub-interval of the requested range that has no coverage. get_bars calls the caller-supplied fetch_fn once per gap (further split into IB-safe chunks if the gap is wide).

BarRecord — the namedtuple returned by get_bars:

BarRecord(date, open, high, low, close, volume, wap, bar_count)

date is the original IB date string (e.g. "20260319 09:30:00 US/Eastern").

---

API Reference

BarStore(db_path)

Open (or create) a SQLite cache at db_path. Creates parent directories. Initialises the schema on first open.

from ib.bar_store import BarStore
store = BarStore("historical/gld_5min.db")

---

get_bars(...) → List[BarRecord]

bars = store.get_bars(
    symbol      = "GLD",
    bar_size    = "5 mins",
    what_to_show= "TRADES",
    use_rth     = True,
    start_dt    = datetime(2026, 1, 1, tzinfo=UTC),
    end_dt      = datetime(2026, 3, 1, tzinfo=UTC),
    fetch_fn    = my_fetch_fn,   # callable(start_dt, end_dt) → list of bar objects
    force       = False,         # True → bypass cache, re-fetch and overwrite
)

fetch_fn receives UTC-aware start_dt / end_dt datetimes and must return a list of objects with .date, .open, .high, .low, .close, .volume (and optionally .wap, .barCount). Returning an empty list is valid; coverage is still recorded for the chunk so it won't be re-requested.

force=True treats the full range as a single gap regardless of coverage. Useful after a corporate action or data correction.

Chunking: large gaps are automatically split into IB-safe chunks (e.g. 30 days for 5-min bars, 1 day for 1-sec bars). Each chunk is a separate fetch_fn call.

---

coverage_summary(symbol=None) → List[dict]

Returns one dict per series (optionally filtered by symbol):

for entry in store.coverage_summary(symbol="GLD"):
    print(entry["symbol"],    # "GLD"
          entry["bar_size"],  # "5 mins"
          entry["intervals"], # [("2026-01-01T00:00:00", "2026-03-01T00:00:00")]
          entry["total_bars"])

Key	Type	Description
symbol	str
bar_size	str
what_to_show	str
use_rth	bool
intervals	list of (start, end) str	ISO-8601 UTC strings
total_bars	int	Total rows in the bars table for this series

---

insert_bar(symbol, bar_size, what_to_show, use_rth, bar)

Insert a single bar into the cache. Called automatically by subscribe_live_bars() for every bar received — you rarely need to call this directly.

store.insert_bar(
    symbol       = "GLD",
    bar_size     = "5 mins",
    what_to_show = "TRADES",
    use_rth      = True,
    bar          = bar_data,   # ibapi BarData-like: .date, .open, .high, .low, .close, .volume
)

Silently no-ops on any failure (bad date format, DB error, etc.) so it is safe to call from hot paths like on_bar callbacks.

---

purge(symbol, bar_size, what_to_show, use_rth) → int

Delete all bars and coverage for a series. Returns the number of bar rows deleted. The next get_bars call will re-fetch from IB.

n = store.purge("GLD", "5 mins", "TRADES", True)
print(f"Deleted {n} bars")

---

duration_str(start_dt, end_dt) → str

Module-level helper. Converts a (start, end) datetime pair to an IB durationStr string ("N S|D|W|M|Y"). Use it inside fetch_fn:

from ib.bar_store import duration_str

def fetch_fn(start_dt, end_dt):
    return plugin.get_historical_data(
        contract=ContractBuilder.etf("GLD"),
        end_date_time=end_dt.strftime("%Y%m%d-%H:%M:%S"),
        duration_str=duration_str(start_dt, end_dt),
        bar_size_setting="5 mins",
        what_to_show="TRADES",
        use_rth=True,
    ) or []

---

Database Schema

Two tables, keyed on the series tuple (symbol, bar_size, what_to_show, use_rth):

bars (
    symbol, bar_size, what_to_show, use_rth,
    bar_dt_utc  TEXT PRIMARY KEY component,   -- ISO-8601 UTC
    bar_dt_orig TEXT,                          -- original IB date string
    open, high, low, close REAL,
    volume INTEGER, wap REAL, bar_count INTEGER
)

coverage (
    symbol, bar_size, what_to_show, use_rth,
    start_utc TEXT, end_utc TEXT,              -- ISO-8601 UTC
    fetched_at TEXT
)

WAL journal mode is enabled for concurrent read access.

---

CLI — ibctl.py historical

Fetch bars from IB through the running engine and save them to a persistent BarStore DB. Every fetch is automatically recorded — no extra flags needed.

Default DB path: historical/bars.db (relative to ibctl.py). Change it once with set-db; the path is stored in historical/config.json.

historical fetch

./ibctl.py historical fetch SYMBOL [options]

Options:
  --bar-size "5 mins"      Bar width (default: "1 day")
  --duration "2 D"         How far back: N S|D|W|M|Y (default: "1 W")
  --end YYYYMMDD-HH:MM:SS  End datetime in UTC (default: now)
  --what TRADES            TRADES | MIDPOINT | BID | ASK (default: TRADES)
  --type etf               Contract type: etf (default) | stock | forex
  --no-rth                 Include extended-hours bars

Examples:

./ibctl.py historical fetch GLD
./ibctl.py historical fetch GLD --bar-size "5 mins" --duration "2 D"
./ibctl.py historical fetch EUR --type forex --what MIDPOINT --no-rth

Output:

[OK] Fetched 156 bars for GLD (5 mins, 2 D)

  Date                        Open      High       Low     Close      Volume
  ------------------------------------------------------------------------
  20260318 09:30:00 US/E    446.66    446.73    445.55    446.29     578,843
  ...
  20260319 15:55:00 US/E    426.18    427.07    426.07    426.43     533,196

[DB] Saved 156 bar(s) to /path/to/historical/bars.db  [GLD / 5 mins / TRADES]

historical coverage

./ibctl.py historical coverage [--symbol SYMBOL]

Shows what is cached in the configured DB. No engine connection needed.

DB: /home/ron/claude/ib/historical/bars.db
Symbol   BarSize    What       RTH     Bars  Intervals
--------------------------------------------------------------------------------
GLD      5 mins     TRADES     Y        156  2026-03-18T13:30:00..2026-03-19T21:00:00

historical purge

./ibctl.py historical purge --symbol SYMBOL [--bar-size "5 mins"] [--what TRADES] [--no-rth]

Deletes a series from the configured DB. No engine connection needed.

historical set-db / get-db

./ibctl.py historical set-db /data/bars.db   # persist a new DB path
./ibctl.py historical get-db                 # show current DB path

set-db writes the path to historical/config.json. All subsequent fetch, coverage, and purge calls use that path.

---

Using BarStore in a Plugin

See Plugin Manual §9.1 for the full reference and §9.2 for the get_bars_cached() shortcut.

Preferred pattern — get_bars_cached

Use the built-in PluginBase.get_bars_cached() method instead of constructing a BarStore manually. It routes through the shared historical/bars.db and only calls IB for uncached ranges:

from datetime import datetime, timedelta, timezone

class MyPlugin(PluginBase):
    def start(self) -> bool:
        end_dt   = datetime.now(timezone.utc)
        start_dt = end_dt - timedelta(days=30)
        bars = self.get_bars_cached(
            contract=ContractBuilder.etf("GLD"),
            start_dt=start_dt, end_dt=end_dt,
            bar_size_setting="5 mins",
            what_to_show="TRADES", use_rth=True,
        )
        return True

Live bars with auto-caching — subscribe_live_bars

Every bar received via subscribe_live_bars() is automatically inserted into the shared BarStore — no manual insert_bar calls needed:

def start(self) -> bool:
    self._req = self.subscribe_live_bars(
        contract=ContractBuilder.etf("GLD"),
        on_bar=self._on_bar,
        bar_size_setting="5 mins",
        what_to_show="MIDPOINT",
        use_rth=True,
    )
    return True

Manual pattern (advanced)

from ib.bar_store import BarStore, duration_str
from ib.contract_builder import ContractBuilder
from zoneinfo import ZoneInfo
from datetime import datetime, timedelta

UTC = ZoneInfo("UTC")

class MyPlugin(PluginBase):
    def start(self) -> bool:
        self._store = BarStore(self._base_path / "bars.db")
        return True

    def _get_bars(self, symbol, days=30):
        end_dt   = datetime.now(UTC)
        start_dt = end_dt - timedelta(days=days)

        def fetch(s, e):
            return self.get_historical_data(
                contract=ContractBuilder.etf(symbol),
                end_date_time=e.strftime("%Y%m%d-%H:%M:%S"),
                duration_str=duration_str(s, e),
                bar_size_setting="5 mins",
                what_to_show="TRADES",
                use_rth=True,
            ) or []

        return self._store.get_bars(
            symbol=symbol, bar_size="5 mins",
            what_to_show="TRADES", use_rth=True,
            start_dt=start_dt, end_dt=end_dt,
            fetch_fn=fetch,
        )

---

Paper Test

plugins/paper_tests/paper_test_bar_store/ verifies BarStore end-to-end against a live paper account. Run it with:

python run_paper_tests.py --bar-store

Test	What it verifies
cold_fetch_gld	Cold cache → IB fetched → bars returned + coverage set
cache_hit_gld	Same range second call → no IB fetch (call count stays 0)
gap_fill_gld	Cache middle day; wider request fills surrounding gaps without re-fetching cached portion
gap_in_middle	Cache outer wings; full-range request fetches only the middle hole
force_refetch_gld	force=True always calls IB even when fully cached
coverage_summary	coverage_summary() returns correct symbol, bar_size, interval, bar count
purge_and_refetch	purge() clears cache; next call re-fetches from IB
multi_symbol	GLD and UUP cached independently — no cross-contamination
ohlc_valid	All returned BarRecords have valid OHLC and positive volume