CSV Streaming Application

A Django-based web application that efficiently streams and displays CSV data in real-time using server-sent events (SSE). The application provides a smooth user experience by loading CSV data progressively without overwhelming the browser's memory.

Features

• Real-time CSV data streaming
• Server-sent events (SSE) implementation
• Progressive data loading
• Responsive table display
• Error handling and validation

Prerequisites

• Python 3.8 or higher
• Django 4.2.2

Installation

1. Clone the repository:

   git clone <repository-url>
   cd <project-directory>

2. Create and activate a virtual environment:

   python -m venv venv
   source venv/bin/activate  # On Windows: .\venv\Scripts\activate

3. Install dependencies:

   pip install -r requirements.txt

4. Run migrations:

   python manage.py migrate

Usage

1. Start the development server:

   python manage.py runserver

2. Access the application:

   • Open your browser and navigate to http://localhost:8000/stream/?file=your_file.csv
   • Replace your_file.csv with the name of your CSV file (must be placed in the data directory)

Project Structure

├── csvstreamer/          # Project configuration
├── data/                 # CSV files directory
├── streamer/             # Main application
│   ├── templates/        # HTML templates
│   └── views.py          # Application views
├── manage.py             # Django management script
└── requirements.txt      # Project dependencies

API Documentation

CSV Streaming Endpoint

• URL: /stream/
• Method: GET
• Query Parameters:
  • file: Name of the CSV file to stream (required)
• Response: Server-sent events stream
• Content Type: text/event-stream

Error Handling

• Returns 400 Bad Request if:
  • CSV file parameter is missing
  • Specified CSV file is not found
• Returns appropriate error message in the stream if:
  • File reading encounters an error
  • CSV parsing fails

Development

• Place your CSV files in the data directory
• The application automatically detects CSV headers
• Data is streamed in chunks for optimal performance

Security Considerations

• The application only serves files from the data directory
• File paths are validated to prevent directory traversal
• Debug mode should be disabled in production

License

This project is licensed under the MIT License - see the LICENSE file for details.