Skip to content

Releases: ccjiao/data-scraper-engine

Release list

πŸ”§ v1.1.0 β€” CI, Tests & Pro Package

πŸ”§ v1.1.0 β€” CI, Tests & Pro Package Latest
Latest

Choose a tag to compare

View all tags
@ccjiao ccjiao released this 15 Jun 06:14
v1.1.0
5fb8a1d

What's New in v1.1.0

πŸ”„ CI/CD

  • GitHub Actions CI pipeline (Python 3.10-3.13 matrix)
  • Ruff + Flake8 lint checks
  • Automated build verification

πŸ§ͺ Testing

  • Unit tests for URL router (7 test cases)
  • Unit tests for config loader (3 test cases)
  • All tests passing βœ…

πŸ“¦ Pro Package

  • datapilot-pro v1.0.0 released!
    • Sentiment analysis (94% accuracy)
    • User profiling (4-dimension scoring, water army detection)
    • REST API server for AI tool integration
    • douyinLive Go service management
  • Install: pip install datapilot-pro

πŸ“ Templates

  • Issue templates (bug report + feature request)
  • PR template with checklist
  • CI status badge in README

Quick Start

git clone https://github.com/ccjiao/data-scraper-engine.git
cd data-scraper-engine
pip install -r requirements.txt

# One command to scrape!
python scripts/scraper.py scrape "https://www.lottery.gov.cn/kj/kjlb.html?dlt"

Pro Features

pip install datapilot-pro

# Sentiment analysis
python scripts/scraper.py scrape "https://live.douyin.com/379595210124" --analyze

# REST API
python scripts/scraper.py serve --port 9898
Assets 2
Loading

πŸŽ‰ v1.0.0 β€” First Public Release

πŸŽ‰ v1.0.0 β€” First Public Release

Choose a tag to compare

View all tags
@ccjiao ccjiao released this 15 Jun 06:03
v1.0.0
6c2e8a9

Data Scraper Engine v1.0.0

What's New

  • 🧠 URL Smart Router β€” Drop any URL, auto-detect the data source type
  • πŸ”„ Multi-Protocol β€” HTTP API + WebSocket + Generic Web adapters
  • 🎰 Lottery Data β€” Full historical data for 倧乐透/ζŽ’εˆ—3/δΈƒζ˜Ÿε½©
  • 🎬 Douyin Live β€” Live room data collection (chat, gifts, interactions)
  • 🌐 Generic Web β€” Extract tables, lists, articles from any URL
  • πŸ“Š Structured Output β€” Styled Excel, JSON, CSV
  • πŸ”§ Config-Driven β€” New data sources only need a YAML file

Quick Start

git clone https://github.com/ccjiao/data-scraper-engine.git
cd data-scraper-engine
pip install -r requirements.txt

# One command to scrape!
python scripts/scraper.py scrape "https://www.lottery.gov.cn/kj/kjlb.html?dlt"

Pro Features (Coming Soon)

  • Sentiment analysis (94% accuracy)
  • User profiling & water army detection
  • REST API server for AI tool integration
  • Scheduled tasks & Webhook push
Loading