Scrapper

A web scraping service built with Node.js, Express, and Puppeteer that allows you to extract HTML content from web pages using CSS selectors.

Features

• Web scraping API endpoint
• CSS selector-based content extraction
• Built with TypeScript
• Containerized with Docker
• Development hot reload

API

GET /handle

Scrapes content from a webpage using a CSS selector.

Query Parameters:

• url (required): The URL of the webpage to scrape
• selector (required): CSS selector for the element to extract

Example:

GET /handle?url=https://example.com&selector=.main-content

Response:

{
  "html": "<div class=\"main-content\">...</div>"
}

Getting Started

Prerequisites

• Node.js 18+ (for local development)
• Docker and Docker Compose (for containerized development)

Option 1: Using Docker (Recommended)

Docker is the recommended approach as it handles all Puppeteer dependencies automatically and provides a consistent environment.

1. Clone the repository

   git clone https://github.com/wellsm/scrapper.git
   cd scrapper

2. Create the external network (if it doesn't exist)

   docker network create kwai

3. Start the application

   docker-compose up --build

4. Access the application

   • The server will be available at: http://localhost:7060
   • Test the scraping endpoint: http://localhost:7060/handle?url=https://www.scrapethissite.com/pages/&selector=.page-title

The Docker setup includes:

• Automatic dependency installation
• TypeScript compilation
• All Puppeteer dependencies pre-installed
• No manual system dependency management required

Option 2: Local Development (Node.js)

1. Clone the repository

   git clone https://github.com/wellsm/scrapper.git
   cd scrapper

2. Install dependencies

   npm install

3. Development mode (with hot reload)

   npm run dev

4. Production build and start

   npm run build
   npm start

5. Access the application

   • The server will be available at: http://localhost:3000
   • Test the scraping endpoint: http://localhost:7060/handle?url=https://www.scrapethissite.com/pages/&selector=.page-title

Note: Local development requires installing Puppeteer system dependencies, which can be complex. Consider using Docker for easier setup. Refer to the Puppeteer documentation for system-specific requirements.

Development

Available Scripts

• npm run dev - Start development server with hot reload
• npm run build - Compile TypeScript to JavaScript
• npm start - Start production server

Project Structure

scrapper/
├── src/
│   └── index.ts          # Main application entry point
├── docker-compose.yml    # Docker Compose configuration
├── Dockerfile           # Docker build configuration
├── package.json         # Project dependencies and scripts
├── tsconfig.json        # TypeScript configuration
└── README.md           # This file

Environment

The application uses the following configuration:

• Port: 3000 (local), 7060 (Docker external)
• Timezone: America/Sao_Paulo
• Node.js version: 22 (in Docker)

Troubleshooting

Common Issues

1. Port already in use

   • For Docker: Change the external port in docker-compose.yml if 7060 is occupied
   • For local development: The default port 3000 can be modified in src/index.ts
   • Check if another application is using the port: lsof -i :3000 or lsof -i :7060

2. Puppeteer issues (Local Development)

   • Recommended: Use Docker to avoid Puppeteer setup complexity
   • For local setup, ensure all system dependencies are installed:
     • On Linux: sudo apt-get install -y gconf-service libasound2 libatk1.0-0 libcairo-gobject2 libdrm2 libgtk-3-0 libnspr4 libnss3 libx11-xcb1 libxcomposite1 libxcursor1 libxdamage1 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6
     • On macOS: Usually works out of the box
     • On Windows: May require additional setup, Docker recommended

3. Docker issues

   • Ensure the kwai network exists: docker network create kwai
   • Check if other containers are using the same network
   • If build fails, try: docker-compose down && docker-compose up --build

4. TypeScript compilation errors

   • Run npm run build to check for compilation issues
   • Ensure TypeScript version compatibility

Logs

To view application logs:

# Docker
docker-compose logs -f

# Local development
# Logs will appear in the terminal where you ran npm run dev