Skip to content

denv3rr/clear

BranchesTags

Repository files navigation

Python Top Language Repo Size GitHub Created At Last Commit Issues Website



Warning

Work in progress.

Overview

A portfolio management, analytics, and global tracking platform.

Features

  • Client and account management
  • Portfolio analytics + regime modeling
  • Market dashboard and macro snapshots
  • Global flight and maritime tracking
  • Intel and news aggregation
  • Reports and exports (CLI + web)
  • Web API with streaming trackers
More Details

Some features still under development

  • Lot-aware holdings and position-level analysis
  • Regime transition matrices, CAPM, and derived analytics
  • Multi-interval views with paging and cached snapshots
  • Tracker heat/volatility metrics and relevance tagging
  • Weather and conflict reporting with caching and exports
  • Health-aware sources with retry/backoff handling

Quick Start

git clone git@github.com:denv3rr/clear.git --depth 1
cd clear
python clearctl.py start

Web App (API + UI)

python clearctl.py start
python clearctl.py stop

If npm is not found, install Node.js first (includes npm) and retry:

Launcher Commands

# Start API + web UI (foreground; Ctrl+C stops)
python clearctl.py start

# Start API only (no web UI)
python clearctl.py start --no-web

# Start the web stack (API + UI)
python clearctl.py web

# Start the CLI
python clearctl.py cli

# Run in background
python clearctl.py start --detach

# Stop background services
python clearctl.py stop

# Status + health
python clearctl.py status

# Logs (last 200 lines)
python clearctl.py logs

# Doctor (deps + port + health checks)
python clearctl.py doctor

Convenience wrappers:

  • Windows PowerShell: .\clear.ps1 start
  • macOS/Linux: ./clear.sh start
  • Windows PowerShell: .\clear.ps1 web / .\clear.ps1 cli
  • Windows CMD: clear web / clear cli
  • macOS/Linux: ./clear web / ./clear cli

Testing

python -m pytest
cd web
npm run test:e2e

Service templates for always-on usage: docs/platform_services.md

Local Reports (Offline)

Generate a client report without any model installed:

python -m modules.reporting.cli --client-id <CLIENT_ID> --format md

Optional output file:

python -m modules.reporting.cli --client-id <CLIENT_ID> --format md --output data/client_weekly_brief.md

Health check:

python -m modules.reporting.cli --health-check

Startup safety checks:

  • run.py performs a syntax compile pass across core modules and warns if config/settings.json is invalid JSON.
  • run.py warns if data/clients.json fails schema checks (client_id/name, account list shape, holdings/lot types).
  • Legacy lot timestamps are normalized to ISO-8601 on startup (and can be forced via Settings -> Diagnostics -> Normalize Lot Timestamps).

Local Model Setup

Note

Incomplete

For model-enhanced reports, install Ollama (recommended) or run a local HTTP server (llama.cpp, vLLM) and run with --use-model:

python -m modules.reporting.cli --client-id <CLIENT_ID> --format md --use-model --model-id llama3

Install instructions:

  • Windows (Scoop): scoop install ollama
  • macOS (Homebrew): brew install ollama
  • Linux: curl -fsSL https://ollama.com/install.sh | sh

Local HTTP servers:

  • llama.cpp server with OpenAI-compatible endpoints (/v1/chat/completions)
  • vLLM or similar local endpoint

Testing

# Run all tests
python -m pytest

Web smoke tests (requires Node/npm):

cd web
npm install
npx playwright install
npm run test:e2e

Optional: validate Playwright browsers via the launcher:

python clearctl.py doctor --web-tests

Configuration

Finnhub API Key

Although the basic Finnhub.io service is free, some international exchanges require a paid plan.

  1. Create a free account: https://finnhub.io/register
  2. Get your API key: https://finnhub.io/dashboard
  3. Copy .env.example to .env
  4. Add your key:
# API KEYS
FINNHUB_API_KEY=your_api_key_here

Warning

Do not commit any .env files.

Web API Key (Optional)

If you set CLEAR_WEB_API_KEY, the API enforces X-API-Key on all routes and the launcher forwards it to the UI as VITE_API_KEY for local auth.

CLEAR_WEB_API_KEY=your_local_key

Environment Variables

OpenSky is the only flight feed right now; configure OAuth credentials in .env for new accounts (legacy basic auth still works for older accounts).

Variable Purpose Used By
FINNHUB_API_KEY Enables Finnhub symbol/quote lookups. Market Feed, Settings
OPENSKY_CLIENT_ID OpenSky OAuth client id (new accounts). Global Trackers
OPENSKY_CLIENT_SECRET OpenSky OAuth client secret (new accounts). Global Trackers
OPENSKY_USERNAME OpenSky basic auth username (legacy accounts). Global Trackers
OPENSKY_PASSWORD OpenSky basic auth password (legacy accounts). Global Trackers
OPENSKY_BBOX OpenSky bounding box (minLat,minLon,maxLat,maxLon). Global Trackers
OPENSKY_EXTENDED Request OpenSky category metadata (1/0). Global Trackers
OPENSKY_ICAO24 Comma-separated ICAO24 filter list. Global Trackers
OPENSKY_TIME Unix timestamp to request historical state vectors. Global Trackers
SHIPPING_DATA_URL Shipping feed endpoint for vessel tracker. Global Trackers
CLEAR_INCLUDE_COMMERCIAL Include commercial flights when set to 1 (default off). Global Trackers
CLEAR_INCLUDE_PRIVATE Include private flights when set to 1 (default off). Global Trackers
CLEAR_GUI_REFRESH GUI tracker refresh seconds (default 10). GUI Tracker
CLEAR_GUI_PAUSED Start GUI tracker paused when 1. GUI Tracker
CLEAR_WEB_API_KEY Enforces API key auth + forwards to UI as VITE_API_KEY. Web API, Web UI

Flight operator metadata can be extended by copying config/flight_operators.example.json to config/flight_operators.json.

AI Synthesis

Note

Incomplete

AI synthesis is configured in config/settings.json under the ai key:

Key Purpose Default
reporting.ai.enabled Toggle report model usage. true
reporting.ai.provider auto, ollama, local_http, or rule_based. auto
reporting.ai.model_id Local model identifier. llama3
reporting.ai.endpoint Local HTTP LLM endpoint for /v1/chat/completions. http://127.0.0.1:11434
reporting.ai.timeout_seconds Per-request timeout in seconds. 15

The top-level ai.* settings remain for other synthesis features; reporting reads reporting.ai first and falls back to ai.* if missing.

Tools Settings

Tools settings live in config/settings.json under the tools key:

Key Purpose Default
tools.perm_entropy_order Permutation entropy order (m). 3
tools.perm_entropy_delay Permutation entropy delay (tau). 1

These can be adjusted in Settings -> Tools. Tools interval selection persists per client; change it from the Tools menu when needed.

Scroll Text Settings

Scroll text settings live in config/settings.json under display.scroll_text:

Key Purpose Default
display.scroll_text.prompt.speed Prompt scroll speed. 8.0
display.scroll_text.prompt.band_width Prompt highlight band width. 3
display.scroll_text.prompt.trail Prompt trail length. 0
display.scroll_text.prompt.highlight_style Prompt highlight style. bold bright_white
display.scroll_text.prompt.base_style Prompt base style. dim
display.scroll_text.warning.speed Warning scroll speed. 8.0
display.scroll_text.warning.band_width Warning highlight band width. 3
display.scroll_text.warning.trail Warning trail length. 0
display.scroll_text.warning.highlight_style Warning highlight style. bold bright_white
display.scroll_text.warning.base_style Warning base style. dim

These can be adjusted in Settings -> Display & UX -> Scroll Text.

Runtime Files (Generated)

Path Purpose
data/intel_news.json Cached RSS news items for reports.
data/news_health.json RSS feed health + backoff state.
data/ai_report_cache.json Cached AI synthesis outputs.
data/clients.json Local client/account data (legacy lot timestamps auto-normalized to ISO-8601).
config/settings.json Runtime settings saved by the Settings module.
data/*.md, data/*.csv, data/*.pdf, exports/, reports/ Generated exports (ignored by git).
.env, data/*.json, config/*local*.json Personal/runtime data (ignored by git).

Modules

  • Client Manager: client profiles, accounts, holdings, lots, and tax settings
  • Market Feed: macro dashboard, tickers, and interval switching
  • Financial Tools: analytics, CAPM metrics, regime modeling
  • Settings: API config, device info, preferences

Stack

Core Runtime

Python PySide6 Pandas NumPy Rich Requests HTTPX python-dotenv psutil Cryptography

Web Platform

FastAPI Uvicorn React TypeScript React Router Node.js Vite Tailwind CSS React Markdown Remark GFM Lucide

Visualization

MapLibre Leaflet Recharts Plotly TanStack Table Framer Motion

Data Sources

Finnhub Yahoo Finance OpenSky

Testing

PyTest Playwright

Data Sources

  • Finnhub: symbols and quotes (optional - see configuration)
  • Yahoo Finance: historical data and macro snapshots
  • OpenSky: flight tracking feed (only source right now; OAuth for new accounts, legacy basic auth optional)
  • Open-Meteo: weather signals for reports
  • GDELT: conflict signals for reports (RSS fallback)
  • RSS News: CNBC Top/World, MarketWatch, BBC Business (cached, health-aware)

Financial Methods

Formulas and references

All formulas are shown in plain text for consistent rendering. Inline citations point to primary or canonical references.

Core returns and risk metrics

Metric Formula Notes Source
Simple return r_t = (P_t / P_{t-1}) - 1 Price-based simple return series. SEC glossary example
Annualized mean mu_annual = mean(r) * A A inferred from timestamp spacing. CFA curriculum (return annualization)
Annualized volatility sigma_annual = std(r) * sqrt(A) Sample std with ddof=1. CFA curriculum (risk measures)
Sharpe ratio S = (mean(r_p) - r_f) / std(r_p) Uses annualized scaling. Investor.gov Sharpe
Sortino ratio `S = (mean(r_p) - r_f) / std(r_p r_p < r_f)` Downside deviation uses returns below r_f.
Max drawdown DD_t = (V_t / peak(V)) - 1, MDD = min(DD_t) Uses cumulative value series. CFA curriculum
Historical VaR VaR_q = quantile(r, 1 - q) Empirical quantile. Investor.gov VaR
CVaR (ES) `CVaR_q = mean(r r <= VaR_q)` Tail average beyond VaR.
VaR 99% VaR_0.99 = quantile(r, 0.01) Historical 99% quantile. Investor.gov VaR
CVaR 99% `CVaR_0.99 = mean(r r <= VaR_0.99)` 99% tail average.
EWMA volatility var_t = lambda * var_{t-1} + (1-lambda) * r_t^2 Lambda defaults to 0.94. RiskMetrics 1996
Concentration (HHI) HHI = sum(w_i^2) Used for sector concentration. Herfindahl-Hirschman Index

Factor and benchmark metrics

Metric Formula Notes Source
Beta beta = cov(r_p, r_m) / var(r_m) Sample covariance and variance. CAPM (Sharpe 1964)
CAPM expected return E[r_p] = r_f + beta * (E[r_m] - r_f) Used for alpha derivation. CAPM (Sharpe 1964)
Alpha alpha = r_p - (r_f + beta * (r_m - r_f)) Annualized using A. CAPM (Sharpe 1964)
R-squared R2 = corr(r_p, r_m)^2 Correlation of portfolio vs benchmark. CFA curriculum
Tracking error TE = std(r_p - r_b) Std of active returns. CFA curriculum
Information ratio IR = mean(r_p - r_b) / TE Active return per active risk. CFA curriculum
Treynor ratio T = (mean(r_p) - r_f) / beta Systematic-risk adjusted return. CFA curriculum
M2 (Modigliani) M2 = r_f + ((mean(r_p) - r_f) / std(r_p)) * std(r_m) Converts Sharpe to benchmark vol. CFA curriculum

Pattern and signal methods (toolkit suite)

Method Formula / Procedure Interpretation Source
FFT spectrum FFT(x_t - mean(x_t)) then power ` X(f) ^2`
CUSUM change points S_t = max(0, S_{t-1} + x_t - mu - k) Flags shifts in mean. NIST/SEMATECH CUSUM
Motif similarity ` z_t - z_i
Shannon entropy H = -sum(p_i * log2 p_i) Histogram-based return entropy; higher = more uniform outcomes, lower = concentrated outcomes. Shannon 1948
Permutation entropy H = -sum(p_i * log2 p_i) / log2(m!) Ordinal pattern entropy (order m, delay tau); higher = more complex ordering. Toolkit uses m=3, tau=1. Bandt & Pompe 2002
Hurst exponent H = 2 * slope(log(lag), log(tau)) <0.5 mean-revert, >0.5 trend. Hurst 1951

Assumptions and data handling

  • Returns use simple price returns; portfolio values are built from holdings and adjusted prices.
  • Annualization factor A is inferred from average timestamp spacing (seconds per year / mean delta).
  • Risk-free rate defaults to 4% annual for risk metrics unless a future config override is added.
  • CAPM and regime models fall back to longer history when samples are insufficient; below thresholds the toolkit reports "Not Assessed".
  • Pattern methods normalize series where required (mean-centering or z-scores) and enforce minimum window sizes.

Validation

  • Toolkit metric tests live in tests/test_toolkit_metrics.py.
  • Market intel news filter tests live in tests/test_intel_news_filters.py.
  • Report synthesis tests live in tests/test_report_synth.py.
  • News emotion analysis filters to fresh items (default 4h) and ignores stale headlines.

Sources Index

Metrics and modeling

Market data and platforms

News sources (current defaults)

Futures and derivatives references

Rates and macro references



Disclaimer

All content is provided 'as is' without warranty; users should consult with a qualified professional before making any investment or financial decisions.

This project is for informational use and does not provide financial, legal, or tax advice.

No information presented should be construed as an offer to buy or sell securities. The author assumes no liability for financial losses or decisions made based on the contents of this project.

About

Books • Analytics • Market Feeds • Logistics Trackers

seperet.com

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Languages