DripDrop 🌊

Progressive JSON streaming for Python web APIs

DripDrop brings the power of Progressive JSON to Python, allowing your APIs to stream data as it becomes available rather than waiting for everything to load before sending a response.

💡 Inspiration

This project was inspired by Dan Abramov's excellent article on Progressive JSON. Dan's exploration of how React Server Components stream data progressively sparked the idea to bring similar capabilities to Python web APIs. All credit for the core Progressive JSON concept goes to Dan and the React team's innovative work on streaming UI and data.

💡 About the name

DripDrop is a play on the word "drip" and "drop". It's a reference to the fact that the data is streamed as it becomes available, rather than waiting for everything to load before sending a response. Like a coffee drips through a filter, the data is streamed as it becomes available.

🎯 Why DripDrop?

Traditional JSON APIs have a fundamental limitation: the client can't do anything until the server sends the last byte. If your API needs to fetch data from multiple sources, slow operations block everything else.

# Traditional approach - everything waits for the slowest part
{
  "header": "Welcome to my blog",        # Ready in 10ms
  "post": {...},                        # Ready in 100ms  
  "comments": [...],                    # Ready in 2000ms ⏳
  "footer": "Hope you like it"          # Ready in 10ms, but waits!
}

DripDrop solves this by streaming data breadth-first as it becomes available:

# Progressive approach - fast parts don't wait for slow parts
{"header": "$1", "post": "$2", "footer": "$3"}     # Initial structure
{"placeholder_id": "$1", "data": "Welcome to my blog"}  # Header streams in
{"placeholder_id": "$3", "data": "Hope you like it"}    # Footer streams in
{"placeholder_id": "$2", "data": {...}}                 # Post streams in when ready

⭐️ Demo

DripDrop includes a Makefile with convenient commands for development and testing:

Setup

# Create virtual environment
make venv

# Install dependencies
make install

Running the Server

# Start the example server (runs on http://localhost:8000)
make run

Testing Endpoints

Test the progressive JSON streaming with these curl commands:

# Test simple endpoint
make request-simple

# Test nested async operations
make request-nested

# Test POST endpoint
make request-post

These commands include the proper Accept: application/x-progressive-json header to receive progressive streaming. Without this header, endpoints return traditional JSON.

🚀 Quick Start

Installation

pip install -r requirements.txt

Basic Usage

from dripdrop import DripDrop
from fastapi import Request
import asyncio

app = DripDrop()

async def load_user_data():
    await asyncio.sleep(1)  # Simulate slow database call
    return {"name": "Alice", "role": "admin"}

async def load_posts():
    await asyncio.sleep(2)  # Simulate even slower call
    return [{"title": "Hello World"}, {"title": "Progressive JSON"}]

@app.progressive_route("/dashboard", methods=["GET"])
async def get_dashboard(request: Request):
    return {
        "user": load_user_data(),      # Will stream when ready (~1s)
        "posts": load_posts(),         # Will stream when ready (~2s)
        "timestamp": "2024-01-01"      # Available immediately
    }

Client Usage

No client implementation is provided yet, but client implementation should be able to put together the progressive JSON response from the server.

📖 How It Works

DripDrop implements the Progressive JSON concept described by Dan Abramov:

1. Initial Structure: Returns the complete object structure with placeholders ($1, $2, etc.) for async operations
2. Progressive Resolution: Streams resolved data as JSON Lines, replacing placeholders when async operations complete
3. Order Independence: Fast operations don't wait for slow ones - everything streams as soon as it's ready

Stream Format

DripDrop uses JSON Lines format with this structure:

{"placeholder_id": null, "data": {...}, "is_initial": true, "is_final": false, "error": null}
{"placeholder_id": "$1", "data": "resolved value", "is_initial": false, "is_final": false, "error": null}
{"placeholder_id": null, "data": null, "is_initial": false, "is_final": true, "error": null}

🔧 API Reference

DripDrop

Extends FastAPI with progressive JSON capabilities.

app = DripDrop()

@app.progressive_route(path, **kwargs)

Decorator that adds progressive streaming support to any endpoint.

Important: The endpoint function must accept a Request parameter for content negotiation to work.

@app.progressive_route("/api/data", methods=["GET", "POST"])
async def my_endpoint(request: Request, ...):
    return {
        "sync_data": "available immediately",
        "async_data": some_async_function()
    }

Content Negotiation

• Progressive: Request with Accept: application/x-progressive-json → Returns streaming JSON Lines
• Traditional: Any other Accept header → Returns fully resolved JSON (waits for all async operations)

🏗️ Advanced Examples

Nested Async Operations

async def load_user_profile(user_id):
    return {
        "basic_info": load_basic_info(user_id),     # Fast
        "preferences": load_preferences(user_id),   # Medium
        "activity_feed": load_activity(user_id),    # Slow
    }

@app.progressive_route("/profile/{user_id}")
async def get_profile(request: Request, user_id: str):
    return {
        "profile": load_user_profile(user_id),
        "metadata": {"timestamp": time.now()}
    }

Error Handling

async def might_fail():
    if random.random() < 0.5:
        raise Exception("Simulated failure")
    return "Success!"

@app.progressive_route("/risky")
async def risky_endpoint(request: Request):
    return {
        "safe_data": "Always works",
        "risky_data": might_fail()  # Will stream error if it fails
    }

⚠️ Important Limitations

🚧 This is a Proof of Concept

DripDrop is an incomplete, simplified implementation designed to demonstrate Progressive JSON concepts. It's not production-ready and has several limitations:

Request Parameter Required

The request: Request parameter is mandatory for progressive routes due to content negotiation:

# ✅ Correct
@app.progressive_route("/api/data")
async def good_endpoint(request: Request):
    return {"data": "value"}

# ❌ Won't work - no content negotiation
@app.progressive_route("/api/data") 
async def bad_endpoint():
    return {"data": "value"}

Client Implementation Required

DripDrop requires custom client code that understands the progressive JSON format. Standard HTTP clients will receive JSON Lines that need special parsing.

Known Issues

• Error handling is basic
• No built-in client libraries
• Limited testing
• No production optimizations
• Memory usage not optimized for large payloads

🎯 Client Implementation Guide

To build a client that works with DripDrop:

1. Check Content Type: Look for application/x-progressive-json
2. Parse JSON Lines: Each line is a separate JSON object
3. Handle Initial Structure: First chunk has is_initial: true
4. Replace Placeholders: When placeholder_id is present, replace that placeholder with the new data
5. Watch for Completion: Stream ends when is_final: true

See the JavaScript example above or implement similar logic in your preferred language.

🤝 Contributing

This project is experimental! Contributions, ideas, and feedback are welcome:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Submit a pull request

📚 Learn More

• Progressive JSON by Dan Abramov - The original article that inspired this implementation
• React Server Components - Real-world usage of progressive streaming
• JSON Lines - The streaming format used by DripDrop

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

📝 Notes

Parts of this code were generated by AI (claude-4-sonnet) and then groomed by hand.

---

Remember: This is a proof of concept demonstrating Progressive JSON streaming. Use at your own risk and consider the limitations before using in production!