🔥 Fireaway Supply Price Monitoring System

Automated web scraping system for monitoring product prices on fireawaysupply.co.uk with daily execution, change detection, and Slack notifications.

🎯 Features

✅ Async Playwright browser automation
✅ ~500 products across 9 categories
✅ Price change detection with historical tracking
✅ Stock availability monitoring
✅ Slack Block Kit notifications
✅ CSV exports (daily snapshots + changes)
✅ AWS S3 backup (optional)
✅ Docker support with multiple execution modes
✅ Retry logic with exponential backoff
✅ Error screenshots and alerts
✅ Daily scheduling with APScheduler

📋 Requirements

Python 3.11+
Docker (optional, recommended)
Slack webhook URL
AWS S3 credentials (optional)

🚀 Quick Start

Option 1: Docker (Recommended)

Clone and configure:

git clone <repo-url>
cd price-updates
cp .env.example .env
# Edit .env with your credentials

Build Docker image:

chmod +x scripts/*.sh
./scripts/docker-build.sh

Run in scheduled mode (daily at 2 AM):
```
./scripts/docker-run-scheduled.sh
```
Or run once manually:
```
./scripts/docker-run-once.sh
```

Option 2: Local Python

Setup virtual environment:

python3.11 -m venv venv
source venv/bin/activate  # or `venv\Scripts\activate` on Windows

Install dependencies:

pip install -r requirements.txt
playwright install chromium

Configure:

cp .env.example .env
# Edit .env with your Fireaway credentials and Slack webhook

Run once:
```
python src/main.py --mode once
```
Run scheduled:
```
python src/scheduler.py
```

⚙️ Configuration

All configuration via environment variables (.env file):

Required

FIREAWAY_USERNAME - Login username for Fireaway Supply portal
FIREAWAY_PASSWORD - Login password for Fireaway Supply portal

Optional

SLACK_WEBHOOK_URL - Slack webhook for notifications (highly recommended)
AWS_S3_ENABLED=false - Enable S3 backup for CSV files
AWS_S3_BUCKET - S3 bucket name
SCRAPE_SCHEDULE="0 2 * * *" - Cron schedule (default: 2 AM daily)
CONCURRENCY_LIMIT=3 - Number of parallel product scrapes
HEADLESS=true - Headless browser mode (false for debugging)
LOG_LEVEL=INFO - Logging level (DEBUG, INFO, WARNING, ERROR)

See .env.example for all available options.

📁 Project Structure

price-updates/
├── src/
│   ├── scrapers/          # Web scraping modules
│   │   ├── browser.py     # Playwright browser setup
│   │   ├── auth.py        # Authentication logic
│   │   ├── category.py    # Category page scraper
│   │   └── product.py     # Product detail scraper
│   ├── storage/           # Data persistence
│   │   ├── database.py    # SQLite operations
│   │   ├── csv_export.py  # CSV file generation
│   │   └── s3_uploader.py # AWS S3 backup
│   ├── notifications/     # Alert systems
│   │   └── slack.py       # Slack Block Kit notifications
│   ├── utils/             # Utilities
│   │   ├── logger.py      # Logging configuration
│   │   ├── retry.py       # Retry decorator
│   │   └── config.py      # Configuration loader
│   ├── main.py            # Main entry point
│   └── scheduler.py       # APScheduler setup
├── data/
│   ├── database.sqlite    # Historical product data
│   ├── exports/           # CSV export files
│   │   ├── products_*.csv # Daily snapshots
│   │   └── changes_*.csv  # Price/stock changes
│   └── screenshots/       # Error screenshots
├── config/
│   └── categories.json    # Category configuration
├── docker/
│   ├── Dockerfile
│   └── docker-compose.yml
├── scripts/               # Helper scripts
│   ├── docker-build.sh
│   ├── docker-run-once.sh
│   └── docker-run-scheduled.sh
├── tests/                 # Test suite
├── requirements.txt       # Python dependencies
├── .env.example           # Environment template
└── README.md              # This file

🐳 Docker Commands

Scheduled mode (runs continuously with daily execution):

docker-compose -f docker/docker-compose.yml up -d fireaway-scraper-scheduled
docker-compose -f docker/docker-compose.yml logs -f  # View logs
docker-compose -f docker/docker-compose.yml down     # Stop

One-shot mode (manual execution):

docker-compose -f docker/docker-compose.yml --profile manual run --rm fireaway-scraper-once

Test mode (dry run):

docker-compose -f docker/docker-compose.yml --profile test run --rm fireaway-scraper-test

📊 CSV Output

Full Snapshot (products_YYYY-MM-DD_HH-MM-SS.csv):

sku,name,category,price_current,price_original,stock_status,url,image_url,last_checked
FA-001,Product Name,Food,15.99,19.99,in-stock,https://...,https://...,2025-11-06 02:00:00

Changes Report (changes_YYYY-MM-DD_HH-MM-SS.csv):

timestamp,sku,name,change_type,old_value,new_value,price_diff,price_diff_percent
2025-11-06 02:00:00,FA-001,Product Name,price_decrease,19.99,15.99,-4.00,-20.01%

🔔 Slack Notifications

Rich formatted messages with:

📉 Price decreases (with % and absolute difference)
📈 Price increases (with alert emoji)
✅ Stock now available (out-of-stock → in-stock)
❌ Out of stock (in-stock → out-of-stock)
🆕 New products (first-time detection)

Example notification:

🔥 Fireaway Supply - Price Update Alert

📉 Price Decreased
Product Name
£19.99 → £15.99 (-£4.00, -20.01%)
Category: Food | SKU: FA-001

🧪 Testing

Test authentication:

python src/main.py --test-auth

Test Slack notifications:

python src/main.py --test-slack

Run full test suite:

pytest tests/ -v

Test with visible browser (debugging):

HEADLESS=false python src/main.py --mode once

📈 Performance

Execution time: ~15-20 minutes (for ~500 products)
Products monitored: ~500 across 9 categories
Concurrency: 3 parallel product scrapes (configurable)
Memory usage: ~500MB (Playwright browser + data)
Storage growth: ~5-10MB per daily run (SQLite + CSVs)
Network: ~100-200 HTTP requests per run

🛠️ Development

Install dev dependencies:

pip install -r requirements-dev.txt

Code formatting:

black src/ tests/

Linting:

pylint src/

Type checking:

mypy src/

Run tests with coverage:

pytest tests/ --cov=src --cov-report=html

🔐 Security

⚠️ Never commit .env file (added to .gitignore)
🔒 Use Docker secrets in production
🔑 Rotate credentials regularly
🛡️ Limit S3 bucket access with IAM policies
📸 Screenshots may contain sensitive data - secure storage location
🔐 Use environment-specific .env files (.env.prod, .env.dev)

🐛 Troubleshooting

Authentication fails

✅ Check username/password in .env
✅ Try with HEADLESS=false to see browser behavior
✅ Check error screenshots in data/screenshots/
✅ Verify website is accessible and login page hasn't changed

Slack notifications not working

✅ Verify webhook URL format is correct
✅ Test with python src/main.py --test-slack
✅ Check webhook permissions in Slack workspace
✅ Review logs for notification errors

S3 upload fails

✅ Check AWS credentials are valid
✅ Verify bucket exists and region is correct
✅ Ensure IAM user has s3:PutObject permission
✅ Test AWS CLI: aws s3 ls s3://your-bucket-name/

Performance issues

⬇️ Reduce CONCURRENCY_LIMIT (try 1 or 2)
⏱️ Increase REQUEST_DELAY_MS (try 3000-5000)
🧹 Clean up old screenshots and CSV exports
💾 Check disk space for database growth

Products not being scraped

✅ Check config/categories.json - all categories enabled?
✅ Verify website structure hasn't changed (selectors)
✅ Check logs for scraping errors
✅ Try scraping single category first

Docker container exits immediately

✅ Check logs: docker-compose logs fireaway-scraper-scheduled
✅ Verify .env file is mounted correctly
✅ Ensure all required env variables are set
✅ Check for Python syntax errors in logs

📋 Maintenance

Regular tasks

📅 Weekly: Review logs for errors
📅 Monthly: Backup database file
📅 Monthly: Clean up old CSV exports
📅 Quarterly: Update dependencies (pip install -U -r requirements.txt)
📅 Quarterly: Review and optimize database (SQLite VACUUM)

Database maintenance

# Backup database
cp data/database.sqlite data/database_backup_$(date +%Y%m%d).sqlite

# Optimize database
sqlite3 data/database.sqlite "VACUUM;"

# Check database size
du -h data/database.sqlite

Log rotation

# Archive old logs
gzip data/scraper.log
mv data/scraper.log.gz data/logs/scraper_$(date +%Y%m%d).log.gz

🔄 Scaling

For more products

Increase CONCURRENCY_LIMIT (4-6 with good connection)
Adjust REQUEST_DELAY_MS to balance speed vs. politeness
Consider running multiple scrapers for different category groups

For production deployment

Use managed scheduling (AWS EventBridge, Google Cloud Scheduler)
Store database in persistent volume or managed database
Use managed Slack app instead of webhooks
Implement monitoring (Datadog, New Relic)
Add alerting for scraper failures

📊 Monitoring

Key metrics to track

✅ Scrape execution time
✅ Number of products scraped
✅ Number of price changes detected
✅ Error rate
✅ Database size growth
✅ Memory usage

Health checks

# Check last successful run
sqlite3 data/database.sqlite "SELECT MAX(last_checked) FROM products;"

# Count products in database
sqlite3 data/database.sqlite "SELECT COUNT(*) FROM products;"

# Check for recent errors
tail -n 100 data/scraper.log | grep ERROR

🎯 Roadmap

Web dashboard for viewing price history
Email notifications (alternative to Slack)
Price alerts with custom thresholds
Product filtering by category/price range
Historical price charts
API endpoint for querying data
Multi-site support (expand beyond Fireaway)
Machine learning for price prediction

📝 License

MIT License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

🤝 Contributing

Contributions are welcome! Please follow these guidelines:

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

Code style

Follow PEP 8
Use type hints
Add docstrings to functions
Write tests for new features

Commit messages

feat: Add price prediction model
fix: Handle missing product images
docs: Update installation instructions
test: Add integration tests for S3 upload

📧 Support

🐛 Issues: GitHub Issues
💬 Discussions: GitHub Discussions
📖 Documentation: See /docs folder for detailed guides

🙏 Acknowledgments

Playwright - Browser automation
APScheduler - Task scheduling
Slack Block Kit - Rich notifications
SQLite - Embedded database

Made with ❤️ for efficient price monitoring

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
.claude		.claude
config		config
data		data
docker		docker
docs		docs
examples		examples
scripts		scripts
src		src
tests		tests
.dockerignore		.dockerignore
.env		.env
.env.example		.env.example
.gitignore		.gitignore
AUTH_FIX_SUMMARY.txt		AUTH_FIX_SUMMARY.txt
BUILD_COMPLETE.md		BUILD_COMPLETE.md
CLAUDE.md		CLAUDE.md
DOCKER-SETUP-COMPLETE.md		DOCKER-SETUP-COMPLETE.md
DOCKER.md		DOCKER.md
FINAL_SUMMARY.md		FINAL_SUMMARY.md
INSTALLATION.md		INSTALLATION.md
Makefile		Makefile
README.md		README.md
STORAGE_MODULES_SUMMARY.md		STORAGE_MODULES_SUMMARY.md
SUCCESS_REPORT.md		SUCCESS_REPORT.md
TEST_RESULTS.md		TEST_RESULTS.md
compare_both.py		compare_both.py
fast_scraper.py		fast_scraper.py
fresco_fast.py		fresco_fast.py
fresco_scraper.py		fresco_scraper.py
quick_test.py		quick_test.py
requirements-dev.txt		requirements-dev.txt
requirements.txt		requirements.txt
requirements_storage.txt		requirements_storage.txt

coder-r/fireaway-price-monitor

Folders and files

Latest commit

History

Repository files navigation