Repository navigation
WebScraper Pro v1.6.0
New Features
1. Real Market Data Integration
- Yahoo Finance provider (yfinance): historical OHLCV, quotes, batch quotes, symbol search
- Alpha Vantage provider: daily adjusted data, global quotes (API key optional)
- Curated list of 16 popular tickers (stocks, indices, commodities, crypto)
- 5-min in-memory cache for repeated requests
- Market Data tab in UI with fetch/quote/batch operations
2. WebSocket Real-Time Streaming
- Async WebSocket server on port 8766 (websockets library)
- Event broadcasting: analysis_result, log, data_update, market_quote, heartbeat
- Client messages: subscribe, ping, get_methods, get_datasets, get_history, run_analysis
- Start/Stop toggle in Export tab
3. Structured Logging System
- LogManager singleton with console (colored) + rotating file (JSON structured) + ring buffer
- 5MB rotating file logs stored in ~/.webscraperpro/logs/
- Log export to JSON/CSV
- In-memory ring buffer (500 entries) for UI log viewer
- Log stats with level counts
- View/Export/Stats/Clear log controls in Export tab
REST API Additions (7 new endpoints)
- GET /api/v1/market/quote/ - Single quote
- POST /api/v1/market/fetch - Fetch historical data
- POST /api/v1/market/batch-quotes - Batch quote
- GET /api/v1/market/tickers - Popular tickers list
- GET /api/v1/logs/recent - Recent log entries
- GET /api/v1/logs/stats - Log statistics
- POST /api/v1/logs/export - Export logs to file
Improvements
- Total: 76 methods across 16 categories (up from 72/15)
- New category: Market Data (4 methods)
- PyInstaller spec updated with yfinance, websockets, asyncio hiddenimports
Files Added
- core/quant/market_data.py (YahooFinanceProvider, AlphaVantageProvider, MarketDataFeed)
- core/api/websocket_server.py (QuantWebSocketServer)
- core/log_manager.py (LogManager, RingBufferHandler, ConsoleFormatter)
Test Results
- All 76 methods tested successfully
- 16 categories verified
- REST API: health, tickers, logs/stats, logs/recent all 200 OK
- LogManager: ring buffer, JSON/CSV export all working
- Backward compatibility: all v1.5.0 features verified