What's New in v1.1.0
🔄 CI/CD
- GitHub Actions CI pipeline (Python 3.10-3.13 matrix)
- Ruff + Flake8 lint checks
- Automated build verification
🧪 Testing
- Unit tests for URL router (7 test cases)
- Unit tests for config loader (3 test cases)
- All tests passing ✅
📦 Pro Package
- datapilot-pro v1.0.0 released!
- Sentiment analysis (94% accuracy)
- User profiling (4-dimension scoring, water army detection)
- REST API server for AI tool integration
- douyinLive Go service management
- Install:
pip install datapilot-pro
📝 Templates
- Issue templates (bug report + feature request)
- PR template with checklist
- CI status badge in README
Quick Start
git clone https://github.com/ccjiao/data-scraper-engine.git
cd data-scraper-engine
pip install -r requirements.txt
# One command to scrape!
python scripts/scraper.py scrape "https://www.lottery.gov.cn/kj/kjlb.html?dlt"
Pro Features
pip install datapilot-pro
# Sentiment analysis
python scripts/scraper.py scrape "https://live.douyin.com/379595210124" --analyze
# REST API
python scripts/scraper.py serve --port 9898
