Skip to content

Releases: codebygk/crawlens

Release list

CrawLens v1.0.0

CrawLens v1.0.0 Latest
Latest

Choose a tag to compare

View all tags
@codebygk codebygk released this 17 Jan 05:35
v1.0.0
4fbdd23

🚀 CrawLens v1.0.0 - Fast Web Crawler with SEO Analysis

**First stable release of CrawLens ** - A blazing-fast, concurrent web crawler with comprehensive SEO analysis and broken link detection.

✨ What's New

Core Features

  • 🔍 Broken Link Detection - Finds all broken links (404s, 500s, network errors) with source page tracking
  • ⚡ High-Performance Crawling - Concurrent crawling with configurable workers (default: 5)
  • 🤖 robots.txt Support - Respects website crawling rules automatically
  • 🔄 Redirect Chain Detection - Identifies and reports multi-hop redirects
  • 🐌 Slow Page Detection - Flags pages exceeding load time thresholds
  • 👻 Orphaned Page Detection - Finds pages with no internal links pointing to them

SEO Analysis (NEW!)

  • 📊 SEO Score (0-100) - Comprehensive scoring for each page based on:

    • Title tag optimization (length, presence)
    • Meta description optimization (length, presence)
    • Heading structure (H1, H2 tags)
    • Content quality (word count, thin content detection)
    • Image accessibility (alt text)
    • Open Graph & Twitter Card tags
    • Canonical URLs, language attributes, robots meta tags

  • 📈 SEO Summary Statistics

    • Average SEO score across site
    • Duplicate title/description detection
    • Common issues aggregation
    • Pages with critical SEO problems

Smart Features

  • ✅ URL Normalization - Treats /page and /page/ as same URL
  • 🎯 Build Artifact Exclusion - Automatically skips _next/static/, webpack chunks, etc.
  • 🔗 Global Link Deduplication - Checks each unique link only once (massive performance boost)
  • 📊 Comprehensive Statistics
    • Crawl duration tracking
    • Total/unique/duplicate link counts
    • Internal vs external link breakdown
    • Average page load time
    • Data transfer statistics

Export Formats

  • JSON - Complete structured data export
  • XML - Standards-compliant XML export
  • CSV - Excel-friendly tabular format

All formats include full crawl statistics, SEO data, broken links, and issues.

User Experience

  • 🎨 Color-Coded Console Output - Easy-to-read terminal output
  • 📍 Progress Indicators - Real-time crawl progress with worker count
  • 🔊 Verbose Mode - Optional detailed logging (-v flag)
  • ⚙️ Flexible Configuration - Control depth, pages, concurrency, thresholds

📖 Usage

Basic Usage

# Download the binary for your platform from releases
# Make it executable (Linux/Mac)
chmod +x crawlens

# Run a basic crawl
./crawlens -u https://example.com

# With SEO analysis and export
./crawlens -u https://example.com -f report.json

# Verbose mode with CSV export
./crawlens -u https://example.com -v -f results.csv

Advanced Usage

# Deep crawl with high concurrency
./crawlens -u https://example.com -d 5 -p 500 -c 20

# Check external links too
./crawlens -u https://example.com -e

# Find slow pages (>5 seconds)
./crawlens -u https://example.com -s 5000

# Full SEO audit with all options
./crawlens -u https://example.com -d 10 -p 1000 -c 15 -e -f audit.json -v

🎯 Command-Line Options

Short Long Description Default
-u --url Starting URL to crawl (required) -
-d --depth Maximum crawl depth 3
-p --pages Maximum pages to crawl 100
-c --concurrency Number of concurrent requests 5
-e --external Check external links false
-s --slow Slow page threshold in milliseconds 3000
-f --file Export report (.json, .xml, .csv) -
-v --verbose Enable verbose output false

📊 What Gets Analyzed

Link Analysis

  • Broken internal links (404s, 500s, timeouts)
  • Broken external links (optional)
  • Redirect chains (301/302 paths)
  • Orphaned pages
  • Internal reference counts

SEO Analysis

  • ✅ Title tags (presence, length 30-60 chars)
  • ✅ Meta descriptions (presence, length 50-160 chars)
  • ✅ H1 tags (exactly 1 per page)
  • ✅ H2 tags (content structure)
  • ✅ Content length (min 300 words)
  • ✅ Image alt text (accessibility)
  • ✅ Canonical URLs
  • ✅ Language attributes
  • ✅ Open Graph tags (social sharing)
  • ✅ Twitter Card tags
  • ✅ Robots meta tags

Performance Metrics

  • Page load times
  • Total bytes transferred
  • Average page load time
  • Slow page detection

🏆 Why CrawLens?

  • Fast - Concurrent crawling with Go's goroutines
  • Smart - Automatic deduplication, normalization, exclusions
  • Complete - Links + SEO + Performance in one tool
  • Actionable - Clear issues with specific recommendations
  • Flexible - JSON/XML/CSV export for any workflow
  • Free & Open Source - MIT licensed

🐛 Known Issues

None!

📝 Full Changelog

Added

  • Initial release with core crawling functionality
  • Broken link detection with source tracking
  • SEO analysis with 0-100 scoring
  • Redirect chain detection
  • Slow page detection
  • Orphaned page detection
  • URL normalization (trailing slash handling)
  • Global link de-duplication
  • Build artifact exclusion patterns
  • Multiple export formats (JSON, XML, CSV)
  • Comprehensive statistics tracking
  • Color-coded console output
  • Verbose logging mode
  • Short and long command-line flags
  • robots.txt support
  • Rate limiting and politeness features

Technical Details

  • Written in Go 1.21+
  • Dependencies: goquery, robotstxt
  • Supports Linux, macOS, Windows
  • Single binary, no runtime dependencies

🙏 Credits

Built with ❤️ using:

📄 License

MIT License - See LICENSE file for details

Full Documentation: README.md
Report Issues: GitHub Issues
Contribute: CONTRIBUTING.md

Assets 8
Loading