Skip to content

Home

Jump to bottom
Martin edited this page Mar 12, 2026 · 11 revisions

Crawrix - Technical Documentation of Web Application

Welcome to the official documentation for Crawrix!
This project is a full-stack web application consisting of a React ⚛️ frontend and backend on Python 🐍 + Flask 🍶.
Local development server used — Vite ⚡.
The project is divided into components.
Frontend code and server connections are written in TSX 🧩 format.
For formatting and code style, Prettier 🎨 and ESLint 🔍 are used.
tsconfig.json 🛠️ is configured for correct project compilation.

📋 Table of Contents

  • Mission
  • Installation Guide
  • Backend Setup
  • Frontend Setup
  • API Documentation
  • Endpoints
  • Usage
  • Backend Requirements
  • Updates

🎯 Mission

The name Crawrix was not chosen by chance; it reflects the essence of the application - to scan entered queries and provide links to them as a response. The goal of this project is simple - to help people find what they need here and now. This tool is useful for both standard users, SEO analysts and web developers.

What is the main difference from a regular search engine? In a regular search engine, you cannot enter a query consisting of different unrelated query words. In Crawrix, the right site will be selected for each keyword. And this is not a service like "Search Engine SEO" such as SEOmoz or Zemrush. These services are aimed at professionals. The Crawrix service is aimed at a standard user who needs to quickly find the necessary information.

Naturally, this service will continue to develop, new features and new service solutions will be added. The design of the application is convenient, adaptive for mobile devices. The current code can be modified but under the name of my author (Martin Daniels). Below you will find detailed information about installing all the necessary application dependencies and detailed information about past updates. The updates are also listed in the Changelog in the README file of the main project repository.

Sincerely, Fullstack Web Developer Martin Daniels. 2025.

Service is mobile-friendly 📱, has a default dark theme 🌙.

⚙️ Installation Guide

🔧 Prerequisites

  • React ⚛️, Vite ⚡, TypeScript 🧩 (Node.js 🟢, npm 📦)
  • Python 🐍 3.13.1, Flask 🍶
  • npm 📦, pip 🔧

🖥️ Backend Setup (Flask 🍶)

Step 1: Clone the repository

git clone https://github.com/Martin13025/Crawrix.git
cd Crawrix

Step 2: Install backend dependencies

cd server
pip install -r requirements.txt

Step 3: Run backend server

cd server
python app.py

Server runs at http://localhost:5000.

🖌️ Frontend Setup (React ⚛️ + Vite ⚡ + TypeScript 🧩)

Step 1: Install frontend dependencies

cd client
npm install

Step 2: Run frontend

npm run dev

Frontend runs at http://localhost:3000.

📚 API Documentation

🔎 Overview

The API parses links based on keywords (via POST request).
It uses DuckDuckGo 🦆 and optionally Yahoo 🟣 (aggressive mode). {Aggressive Mode has been deleted}

Warning

Note: Not intended for heavy production use.

🚀 Endpoints

POST /parse

Description: Parses links for provided keywords.

Request Body:

{
  "keywords": ["example keyword"],
  "lang": "en",
  "aggressive_mode": true
}

Response:

[
  {
    "keyword": "example keyword",
    "links": [
      "https://examplelink1.com",
      "https://examplelink2.com"
    ]
  }
]

Status Codes:

  • 200 OK: Success
  • 400 Bad Request: Invalid input
  • 500 Internal Server Error: Server error

Warning

Changes on external sites (DuckDuckGo, Yahoo) may affect parsing.

GET /api/status

Description: Returns service status and uptime.

Response:

{
  "status": "active",
  "uptime": "10 days 5 hours"
}

📖 Usage Example

1. Send a POST request:

curl -X POST http://yourdomain.com/parse \
-H "Content-Type: application/json" \
-d '{"keywords": ["example search"], "lang": "es", "aggressive_mode": true}'

2. Expected Response:

[
  {
    "keyword": "example search",
    "links": [
      "https://example1.com",
      "https://example2.com"
    ]
  }
]

🖥️ Usage

Start backend:

python app.py

Start frontend:

npm run dev
  • Frontend: http://localhost:3000
  • Backend API: http://localhost:5000

🧩 Backend Requirements

Main Python dependencies:

  • Flask 🍶
  • Flask-Cors 🔥
  • BeautifulSoup4 🍜
  • Requests 🛰️

From requirements.txt:

beautifulsoup4==4.12.3
blinker==1.9.0
certifi==2024.12.14
charset-normalizer==3.4.1
click==8.1.8
colorama==0.4.6
Flask==3.1.0
Flask-Cors==5.0.0
idna==3.10
itsdangerous==2.2.0
Jinja2==3.1.5
MarkupSafe==3.0.2
requests==2.32.3
soupsieve==2.6
urllib3==2.3.0
Werkzeug==3.1.3
gunicorn==21.2.0
Flask-Limiter==3.12

Clone this wiki locally