React + Flask Web Scraper with OpenAI Integration

This project is a full-stack web application that uses React for the frontend and Flask for the backend. The app allows users to submit a URL, scrapes the content, generates questions based on the content using OpenAI's GPT API, and categorizes users based on their answers.

Table of Contents

• Features
• Tech Stack
• Setup Instructions
• Usage
• Project Structure
• API Documentation
• Testing
• Troubleshooting
• Contributing
• License

Features

• Scrapes content from a provided URL.
• Uses OpenAI GPT API to generate dynamic questions based on content.
• Presents questions to the user and categorizes them based on answers.
• Mock responses available for testing without consuming OpenAI quota.

Tech Stack

• Frontend: React, Axios
• Backend: Flask, Flask-CORS, BeautifulSoup, OpenAI API
• Libraries: Python, Node.js
• Third-Party Services: OpenAI GPT API

Setup Instructions

Prerequisites

• Python 3.x
• Node.js and npm
• OpenAI API key

Backend Setup (Flask)

1. Clone the repository and navigate to the flask-backend directory:

   git clone <repo_url>
   cd react-flask-app/flask-backend

2. Create and activate a virtual environment:

   python3 -m venv venv
   source venv/bin/activate  # On Windows use `venv\Scripts\activate`

3. Install dependencies:

   pip install -r requirements.txt

4. Create a .env file in the flask-backend directory and add your OpenAI API key:

   OPENAI_API_KEY=your_openai_api_key
   

5. Start the Flask server:

   python app.py

Frontend Setup (React)

1. Navigate to the react-frontend directory:

   cd ../react-frontend

2. Install dependencies:

   npm install

3. Start the React development server:

   npm start

The React app should now be running at http://localhost:3000, and the Flask API server should be running at http://localhost:5000.

Usage

1. Open the React app in your browser at http://localhost:3000.
2. Enter a URL in the input field and submit.
3. The backend will scrape the content, generate questions, and display them in the app.
4. Select answers to the questions and submit to view your categorized result.

Example Screenshots

Project Structure

react-flask-app/
├── flask-backend/
│   ├── app.py              # Flask API setup and route handlers
│   ├── requirements.txt    # Python dependencies for Flask
│   ├── .env                # Environment variables (not in repo)
│   └── venv/               # Virtual environment (not in repo)
├── react-frontend/
│   ├── src/
│   │   ├── App.js          # Main React component
│   │   └── index.js        # Entry point for React
│   ├── package.json        # Node dependencies for React
└── README.md               # Project documentation

API Documentation

POST /generate-questions

• Description: Accepts a URL, scrapes content, and returns generated questions.
• Request Body:

  {
    "url": "https://example.com"
  }

• Response:

  {
    "questions": [
      {
        "question": "What is your primary interest?",
        "options": ["Technology", "Science", "Literature"]
      }
    ]
  }

POST /categorize

• Description: Accepts user responses and returns a category.
• Request Body:

  {
    "user_response": "Science, Hands-on Practice"
  }

• Response:

  {
    "category": "You are a hands-on science enthusiast."
  }

Testing

Mock Testing

For testing without consuming OpenAI API quota, mock the analyze_with_chatgpt function in app.py as follows:

def analyze_with_chatgpt(content):
    return {
        "questions": [
            {
                "question": "Sample question?",
                "options": ["Option A", "Option B", "Option C"]
            }
        ]
    }

This setup allows testing of the frontend without relying on the OpenAI API.

Troubleshooting

CORS Issues

If you see a CORS error in the console, ensure flask-cors is installed, and CORS(app) is in your app.py file.

OpenAI Quota Errors

If you see "quota exceeded" errors, verify your OpenAI account plan and usage on the OpenAI Usage Page.

Contributing

1. Fork the repository.
2. Create a new branch (git checkout -b feature-branch).
3. Make your changes and commit (git commit -m 'Add feature').
4. Push to the branch (git push origin feature-branch).
5. Open a Pull Request.