Visual Assistant API

A RESTful API that allows users to upload images and have interactive conversations about them. Built with Flask, SQLAlchemy, and OpenAI-compatible response formats. Features streaming responses, conversation history, and persistent storage.

Features

• Image Upload - Upload images with multi-layer validation (extension + content)
• Interactive Chat - Ask questions about uploaded images
• Streaming Responses - Real-time Server-Sent Events (SSE) for chat responses
• Conversation History - Context-aware conversations that remember previous messages
• Persistent Storage - SQLite database with SQLAlchemy ORM
• Caching Layer - In-memory cache for improved performance
• Security - Input validation, secure file handling, SQL injection protection
• Thread-Safe - Handles concurrent requests with proper locking

Prerequisites

• Python 3.8 or higher
• pip or pipenv

Quick Start

Option 1: Using pipenv (Recommended)

# Install pipenv if you don't have it
pip install pipenv

# Install dependencies
pipenv install

# Run the application
pipenv run python app.py

The API will be available at http://127.0.0.1:5000

Endpoints

1. Upload Image

Endpoint: POST /upload

Description: Upload an image file and receive initial analysis

Request:

• Content-Type: multipart/form-data
• Body:
  • file (required): Image file (png, jpg, jpeg, or gif)
  • Maximum file size: 16 MB

2. Chat About Image

Endpoint: POST /chat/<image_id>

Description: Ask questions about an uploaded image (non-streaming)

Request:

• Content-Type: application/json
• URL Parameter: image_id - UUID of the uploaded image
• Body:

{
  "question": "What colors are in this image?"
}

3. Chat About Image (Streaming)

Endpoint: POST /chat-stream/<image_id>

Description: Ask questions about an uploaded image with real-time streaming responses

Request:

• Content-Type: application/json
• URL Parameter: image_id - UUID of the uploaded image
• Body:

{
  "question": "Describe this image in detail"
}

Using Postman

Import the Postman Collection

1. Open Postman
2. Click "Import" in the top left
3. Select "Raw text" tab
4. Copy and paste the collection below
5. Click "Import"

Postman Collection JSON

{
  "info": {
    "name": "Visual Assistant API",
    "description": "API for uploading images and having interactive conversations about them",
    "schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
  },
  "variable": [
    {
      "key": "base_url",
      "value": "http://127.0.0.1:5000",
      "type": "string"
    },
    {
      "key": "image_id",
      "value": "",
      "type": "string"
    }
  ],
  "item": [
    {
      "name": "1. Upload Image",
      "event": [
        {
          "listen": "test",
          "script": {
            "exec": [
              "// Save image_id for subsequent requests",
              "if (pm.response.code === 201) {",
              "    var jsonData = pm.response.json();",
              "    pm.collectionVariables.set('image_id', jsonData.image_id);",
              "    console.log('Image ID saved:', jsonData.image_id);",
              "}"
            ],
            "type": "text/javascript"
          }
        }
      ],
      "request": {
        "method": "POST",
        "header": [],
        "body": {
          "mode": "formdata",
          "formdata": [
            {
              "key": "file",
              "description": "Select an image file (png, jpg, jpeg, or gif)",
              "type": "file",
              "src": []
            }
          ]
        },
        "url": {
          "raw": "{{base_url}}/upload",
          "host": ["{{base_url}}"],
          "path": ["upload"]
        },
        "description": "Upload an image file for analysis. Automatically saves the image_id for use in subsequent requests."
      },
      "response": []
    },
    {
      "name": "2. Chat About Image",
      "request": {
        "method": "POST",
        "header": [
          {
            "key": "Content-Type",
            "value": "application/json"
          }
        ],
        "body": {
          "mode": "raw",
          "raw": "{\n  \"question\": \"What do you see in this image?\"\n}"
        },
        "url": {
          "raw": "{{base_url}}/chat/{{image_id}}",
          "host": ["{{base_url}}"],
          "path": ["chat", "{{image_id}}"]
        },
        "description": "Ask a question about the uploaded image. Non-streaming response."
      },
      "response": []
    },
    {
      "name": "3. Chat About Image (Streaming)",
      "request": {
        "method": "POST",
        "header": [
          {
            "key": "Content-Type",
            "value": "application/json"
          }
        ],
        "body": {
          "mode": "raw",
          "raw": "{\n  \"question\": \"Describe this image in detail\"\n}"
        },
        "url": {
          "raw": "{{base_url}}/chat-stream/{{image_id}}",
          "host": ["{{base_url}}"],
          "path": ["chat-stream", "{{image_id}}"]
        },
        "description": "Ask a question about the uploaded image. Returns streaming Server-Sent Events (SSE) response."
      },
      "response": []
    },
    {
      "name": "4. Chat with History Context",
      "request": {
        "method": "POST",
        "header": [
          {
            "key": "Content-Type",
            "value": "application/json"
          }
        ],
        "body": {
          "mode": "raw",
          "raw": "{\n  \"question\": \"Can you tell me more about that?\"\n}"
        },
        "url": {
          "raw": "{{base_url}}/chat/{{image_id}}",
          "host": ["{{base_url}}"],
          "path": ["chat", "{{image_id}}"]
        },
        "description": "Ask a follow-up question. The API remembers previous conversation context."
      },
      "response": []
    },
    {
      "name": "5. Error Test - Invalid Image",
      "request": {
        "method": "POST",
        "header": [],
        "body": {
          "mode": "formdata",
          "formdata": [
            {
              "key": "file",
              "description": "Upload a non-image file to test validation",
              "type": "file",
              "src": []
            }
          ]
        },
        "url": {
          "raw": "{{base_url}}/upload",
          "host": ["{{base_url}}"],
          "path": ["upload"]
        },
        "description": "Test error handling by uploading a non-image file (e.g., .txt, .pdf)"
      },
      "response": []
    },
    {
      "name": "6. Error Test - Image Not Found",
      "request": {
        "method": "POST",
        "header": [
          {
            "key": "Content-Type",
            "value": "application/json"
          }
        ],
        "body": {
          "mode": "raw",
          "raw": "{\n  \"question\": \"What's in this image?\"\n}"
        },
        "url": {
          "raw": "{{base_url}}/chat/00000000-0000-0000-0000-000000000000",
          "host": ["{{base_url}}"],
          "path": ["chat", "00000000-0000-0000-0000-000000000000"]
        },
        "description": "Test 404 error by using a non-existent image ID"
      },
      "response": []
    },
    {
      "name": "7. Error Test - Missing Question",
      "request": {
        "method": "POST",
        "header": [
          {
            "key": "Content-Type",
            "value": "application/json"
          }
        ],
        "body": {
          "mode": "raw",
          "raw": "{}"
        },
        "url": {
          "raw": "{{base_url}}/chat/{{image_id}}",
          "host": ["{{base_url}}"],
          "path": ["chat", "{{image_id}}"]
        },
        "description": "Test 400 error by sending request without 'question' field"
      },
      "response": []
    }
  ]
}

How to Use the Postman Collection

1. Upload Image (Request #1)

   • Click on body → form-data → click "Select Files" next to the file key
   • Choose an image from your computer
   • Click "Send"
   • The image_id will automatically be saved for subsequent requests

2. Chat About Image (Request #2)

   • Make sure you've uploaded an image first
   • Edit the question in the request body if desired
   • Click "Send"
   • Notice the response references previous conversation if you've chatted before

3. Chat Streaming (Request #3)

   • Same as #2, but returns streaming SSE events
   • You'll see multiple chunks arriving in the response

4. Test Conversation History (Request #4)

   • Send multiple chat requests with different questions
   • Notice how the API references previous conversation context

5. Error Tests (Requests #5-7)

   • Test various error scenarios to see proper error handling

Tech Stack

• Framework: Flask 2.0+
• Database: SQLAlchemy 1.4+ with SQLite
• Image Processing: Pillow 9.0+
• File Security: Werkzeug secure_filename
• Response Format: OpenAI Chat Completions API compatible