Live: https://beyond-chats-olive.vercel.app # BeyondChats Full Stack Assignment Production-ready implementation covering backend scraping, automation with LLM, and a React + Tailwind frontend. ## Overview - Phase 1: Node.js + Express + MongoDB scraper and CRUD APIs - Phase 2: Automation script to search references, scrape content, rewrite with LLM, and save updated articles - Phase 3: React + Tailwind UI to list articles and view details with Original | Updated tabs Live links (placeholders): - Backend API: https://your-backend-host.example.com - Frontend UI: https://your-frontend-host.example.com ## Architecture - Backend (Express): - Article model stores title, content, originalContent, sourceUrl, type, references, createdAt - Scraper fetches the last page of BeyondChats blogs, extracts 5 oldest articles, and stores as type=original - CRUD endpoints for articles - Automation: - Fetch original articles via API, search DuckDuckGo for top references (excluding BeyondChats), scrape main content, call LLM (OpenAI or Gemini) to rewrite, append references, save as type=updated - Frontend (React + Tailwind): - List page shows all articles - Detail page shows source link, created time, tabs for Original and Updated content, and references ## Data Flow 1. Scraper script stores original articles in MongoDB via the model (or REST APIs). 2. Automation script reads originals from API, searches web, scrapes content, calls LLM, saves updated articles via API. 3. Frontend reads from backend APIs to display lists and details. ## Folder Structure ``` FullStack/ ├── backend/ │ └── src/ │ ├── app.js │ ├── server.js │ ├── models/ │ │ └── Article.js │ ├── routes/ │ │ └── articles.js │ ├── services/ │ │ └── scraper.js │ ├── scripts/ │ │ └── run-scrape.js │ └── utils/ │ ├── db.js │ └── errors.js ├── scripts/ │ └── automate.js ├── frontend/ │ ├── index.html │ ├── package.json │ └── src/ │ ├── App.jsx │ ├── main.jsx │ ├── index.css │ ├── pages/ │ │ ├── ArticleList.jsx │ │ └── ArticleDetail.jsx │ └── utils/ │ └── api.js ├── .env.example ├── package.json └── README.md ``` ## Environment Variables Create a `.env` in repository root based on `.env.example`: ``` PORT=5000 MONGODB_URI=mongodb://localhost:27017/beyondchats OPENAI_API_KEY=your_openai_api_key # optional if using OpenAI GEMINI_API_KEY=your_gemini_api_key # optional if using Gemini SERPAPI_KEY=your_serpapi_key # required for Phase 2 automation CLIENT_URL=http://localhost:5173 API_BASE=http://localhost:5000 # used by scripts/automate.js ``` ## Backend: Scripts and APIs Install dependencies: ``` npm install ``` Run backend in development: ``` npm run dev ``` Scrape 5 oldest articles from last page: ``` npm run scrape ``` APIs (base path `/api/articles`): - POST `/api/articles` — create article - GET `/api/articles` — list articles (supports `type`, `search`, `page`, `limit`) - GET `/api/articles/:id` — get by id - PUT `/api/articles/:id` — update by id - DELETE `/api/articles/:id` — delete by id Example curl: ``` curl http://localhost:5000/api/articles?type=original ``` ## Automation Script (Phase 2) Purpose: Read original articles, search references, scrape main content, call LLM to rewrite, save updated. Run: ``` npm run automation ``` LLM Prompt (excerpt): ``` You are an expert technical writer. Rewrite the following article to improve formatting, readability, and SEO. Inspiration: Use the ideas and structure cues from the reference articles, but avoid any plagiarism. Do not copy sentences verbatim. Produce clean HTML with semantic tags (h2/h3, p, ul/ol, strong/em, code where needed). ... Output only HTML for the rewritten content body (no or ). ``` Error Handling: - Network timeouts use axios with 20s limits and user-agent headers - Individual reference scraping failures are logged and skipped - Missing API keys are detected with clear error messages - Backend request failures surface meaningful messages ## Frontend (Phase 3) Move into frontend directory and install dependencies: ``` cd frontend npm install npm run dev ``` Configure backend URL by setting `VITE_API_BASE_URL` in an `.env` file in `frontend/`: ``` VITE_API_BASE_URL=http://localhost:5000 ``` Pages: - Article list page at `/` - Article detail page at `/articles/:id` with tabs for Original | Updated and references ## Deployment - Backend (Render/Railway): - Set environment variables: `PORT`, `MONGODB_URI`, `OPENAI_API_KEY` or `GEMINI_API_KEY` - Start command: `npm start` - Healthcheck: `/health` - Frontend (Vercel/Netlify): - Build: `npm run build` - Environment: `VITE_API_BASE_URL` pointing to your deployed backend ## Best Practices Included - Express security headers via Helmet and CORS - Centralized error handling and logging - Mongoose indexes for performance - Clear separation of concerns (routes, models, services) - Robust scraping heuristics and timeouts - Configurable automation script with optional LLM providers ## Notes - The scraper uses heuristics to locate pagination and article content; adjust selectors if BeyondChats changes its layout. - For stricter CORS in production, set `CLIENT_URL` and limit `origin` in `backend/src/app.js`. # BeyondChats