Web Cloner

Node.js-based tool for cloning complete websites including all pages, assets, and resources using Puppeteer.

Installation

Requirements

• Node.js 16.x or higher
• NPM 7.x or higher

Setup

# Install dependencies
npm install

This will download Puppeteer and Chromium (approximately 170-300MB).

Usage

Clone a Website

node clone.js <URL>

Examples

# Clone a website
node clone.js https://www.example.com

# Or with npm
npm start https://www.example.com

Output Location

Downloaded sites are saved to your Downloads folder:

• macOS: ~/Downloads/<domain-name>/
• Windows: C:\Users\<Username>\Downloads\<domain-name>\

Example: Cloning https://www.toca.social/ creates ~/Downloads/toca.social/

Viewing Cloned Sites

Important: Downloaded HTML files cannot be opened directly by double-clicking. You must use a local server.

Quick Method: Use Helper Script

./serve.sh

This starts a local server automatically. Open your browser to:

http://localhost:8000/index.html

Manual Method: Start Server

Option 1: Python

cd ~/Downloads/<domain-name>
python3 -m http.server 8000

Option 2: PHP

cd ~/Downloads/<domain-name>
php -S localhost:8000

Option 3: Node.js

cd ~/Downloads/<domain-name>
npx http-server -p 8000

Then open: http://localhost:8000/index.html

Why You Need a Server

Modern browsers block JavaScript on file:// protocol due to CORS security. Without a server:

• JavaScript won't execute
• CSS may not load properly
• Interactive features won't work
• Forms won't function

How It Works

1. Navigate to the specified URL
2. Render page with JavaScript execution
3. Download all resources (CSS, JS, images, fonts, videos)
4. Extract internal links from HTML
5. Crawl all discovered pages automatically
6. Convert URLs to local paths
7. Save everything organized by type

What Gets Downloaded

• All HTML pages from the same domain
• CSS stylesheets
• JavaScript files
• Images (jpg, png, svg, webp, avif)
• Fonts (woff, woff2, ttf)
• Videos (mp4, webm)

What Doesn't Get Downloaded

• External domain resources
• Streaming content
• Dynamic API responses
• Content behind login
• Third-party scripts from other domains

Output Structure

Downloads/<domain-name>/
├── index.html
├── about-us.html
├── contact.html
├── [more HTML pages]
└── assets/
    ├── css/
    ├── js/
    ├── images/
    ├── fonts/
    ├── videos/
    └── other/

Files are named based on URL:

• / → index.html
• /about → about.html
• /products/item → products-item.html

Features

• Automatic Crawling: Discovers and downloads all internal pages
• Asset Management: Downloads and organizes all resources
• URL Conversion: Converts absolute URLs to relative paths
• Duplicate Prevention: Tracks visited pages to avoid loops
• Cross-Platform: Works on macOS, Windows, Linux
• Progress Monitoring: Shows browser and download progress

Laravel Migration

For migrating cloned sites to Laravel, see ANALYSIS.md for:

• Detailed migration strategies
• Implementation options
• Cost and timeline estimates
• Technical recommendations

Quick Laravel Setup

# Copy assets to Laravel public folder
cp -r ~/Downloads/<domain-name>/assets public/

# Copy HTML files to views
cp ~/Downloads/<domain-name>/*.html resources/views/

# Rename to .blade.php and update asset paths
# Replace: src="assets/..."
# With: src="{{ asset('assets/...') }}"

Troubleshooting

Browser Launch Error

Error: Failed to launch the browser process

Solution:

npm install puppeteer@latest

Memory Issues

Error: JavaScript heap out of memory

Solution:

node --max-old-space-size=4096 clone.js <URL>

Navigation Timeout

Error: Navigation timeout exceeded

Causes:

• Slow internet connection
• Large page with heavy resources
• Anti-bot protection

Solutions:

• Check internet connection
• Verify URL is accessible
• Try again later

Missing Resources

Issue: Some images or files missing

Causes:

• Lazy loading not triggered
• JavaScript-generated URLs
• CORS restrictions
• CDN authentication

Solution: Re-run cloner or manually download missing files

Limitations

• Does not handle login-protected content
• Cannot download streaming video content
• May miss JavaScript-generated URLs
• Forms won't submit (no backend)
• External API calls will fail offline
• Large sites (1000+ pages) may take hours

Performance

Average download times:

• Small sites (< 50 pages): 5-10 minutes
• Medium sites (50-200 pages): 15-30 minutes
• Large sites (200-500 pages): 30-60 minutes
• Enterprise sites (500+ pages): 1-3 hours

Technical Details

Dependencies

• puppeteer: Browser automation
• axios: HTTP client for downloads
• fs-extra: File system operations
• cheerio: HTML parsing
• path: Path manipulation
• url: URL parsing

Browser Configuration

• Viewport: 1920x1080
• Page timeout: 90 seconds
• Headless: Disabled (visible browser)
• Wait strategy: DOM content loaded

Resource Handling

• Concurrent downloads for speed
• 30-second timeout per resource
• MD5 hash-based filenames (8 chars)
• Organized by file type

Use Cases

Website Migration

• Webflow to Laravel
• WordPress to custom CMS
• Platform migrations

Translation Projects

• Clone original site
• Translate content
• Deploy to new domain

Offline Archive

• Legal compliance
• Historical preservation
• Backup purposes

Development Reference

• Competitor analysis
• Design inspiration
• Feature comparison

License

Provided as-is for educational and development purposes. Users are responsible for compliance with website terms of service and copyright laws.

Disclaimer

This tool is for legitimate purposes only:

• Migrating your own websites
• Archiving sites you own
• Development with proper authorization

Always respect terms of service, robots.txt, and copyright laws.

---

Version: 1.0.0 Node.js: >= 16.0.0 Status: Production Ready

For detailed technical analysis and Laravel migration strategies, see ANALYSIS.md.