Formbricks Engineering Challenge

A backend-only Python CLI system for managing local Formbricks instances, generating realistic seed data using LLMs, and seeding that data via Formbricks APIs.

Why No Frontend?

This is a backend-focused engineering challenge. The system interacts with Formbricks exclusively through its Management and Client APIs, not through any UI. All functionality is exposed through CLI commands for automation and testing purposes.

Why no direct database access?
This project intentionally uses the Formbricks APIs rather than direct database manipulation to:

• Respect Formbricks' business logic and validation rules
• Test the public API surface as a real integration would
• Ensure data integrity through proper API workflows
• Demonstrate proper API client architecture patterns

Project Structure

formbricks-challenge/
├── main.py                     # CLI entrypoint
├── docker-compose.yml          # Formbricks + Postgres orchestration
├── requirements.txt            # Python dependencies
├── .env.example                # Environment configuration template
├── formbricks/                 # Core Python package
│   ├── __init__.py
│   ├── api_client.py           # Formbricks API client (Management + Client)
│   ├── up.py                   # Docker start command
│   ├── down.py                 # Docker stop command
│   ├── generate.py             # LLM-powered data generation
│   └── seed.py                 # API-based data seeding
└── generated_data/             # LLM-generated JSON files (users, surveys, responses)

Prerequisites

• Python 3.8+
• Docker and Docker Compose
• OpenAI API Key (or local Ollama instance)

Setup

1. Install Python dependencies:

   pip install -r requirements.txt

2. Configure environment:

   cp .env.example .env
   # Edit .env with your configuration:
   # - LLM_PROVIDER: 'openai' or 'ollama'
   # - OPENAI_API_KEY: Your OpenAI key (if using OpenAI)
   # - FORMBRICKS_URL: http://localhost:3000
   # - FORMBRICKS_MANAGEMENT_API_KEY: Your Management API key

CLI Commands

All commands follow the pattern: python main.py formbricks <command>

1. Start Environment

Starts Formbricks and Postgres containers in detached mode.

python main.py formbricks up

2. Generate Data

Uses an LLM to generate exactly 10 users, exactly 5 surveys, and at least 1 response per survey.

python main.py formbricks generate

Outputs JSON files to generated_data/:

• users.json: 10 users with name, email, and role (Manager or Owner)
• surveys.json: 5 surveys with title, description, and questions
• responses.json: Survey responses with linked user and survey indices

Validation is strict: If the LLM generates incorrect counts (e.g., 9 users instead of 10), the command fails and no files are saved. This ensures data integrity before seeding.

3. Seed Data

Reads generated JSON files and seeds data into Formbricks via APIs.

python main.py formbricks seed

Seeding process:

1. Creates users via Management API
2. Assigns Manager/Owner roles to users via Management API (MANDATORY step)
3. Creates surveys via Management API
4. Submits responses via Client API (using dynamically mapped IDs)

Role assignment is explicit and mandatory: After each user is created, the system immediately assigns their role via a separate API call. If role assignment fails, the entire seeding process stops to prevent incomplete data states.

4. Stop Environment

Stops and removes containers and volumes.

python main.py formbricks down

Architecture Decisions

API Separation: Management vs Client

This project uses two distinct Formbricks APIs with different purposes and authentication models:

1. Management API (Authenticated)

• Base URL: {FORMBRICKS_URL}/api/v1/management
• Authentication: Requires x-api-key header with Management API key
• Purpose: Administrative operations that modify system state
• Used for:
  • Creating users (POST /users)
  • Assigning roles (PATCH /users/{id}/role)
  • Creating surveys (POST /surveys)

2. Client API (Public/Unauthenticated)

• Base URL: {FORMBRICKS_URL}/api/v1/client
• Authentication: None required (public endpoint)
• Purpose: Public-facing operations like response submission
• Used for:
  • Submitting survey responses (POST /responses)

Why this separation matters:

• Security: Admin operations require authentication, public operations don't
• Correct usage: Mixing these APIs would violate Formbricks' intended security model
• Real-world pattern: This mirrors how a real application would integrate with Formbricks (admin panel uses Management API, public survey form uses Client API)

Explicit Role Assignment

Why roles are assigned via dedicated API calls:

After creating a user, the system makes a separate, explicit API call to assign their role (Manager or Owner). This is not optional.

# 1. Create user via Management API
user = client.create_user({"name": "...", "email": "..."})

# 2. Immediately assign role via Management API (MANDATORY)
client.assign_user_role(user["id"], "Manager")  # or "Owner"

Why this approach:

• No implicit defaults: Roles are never assumed or inferred
• Fail loudly: If role assignment fails, the entire process stops
• Auditability: Each role assignment is logged explicitly
• API-first: Respects Formbricks' API design (roles are a separate concern from user creation)

What happens if role assignment fails:
The seeding process raises an exception and stops immediately. Partial data may exist in Formbricks, requiring a fresh restart (down + up).

LLM Data Validation

The generate command enforces hard constraints before saving any files:

Constraint	Required	What happens if violated
User count	EXACTLY 10	Generation fails, no files saved
Survey count	EXACTLY 5	Generation fails, no files saved
Responses per survey	AT LEAST 1	Generation fails, no files saved
User roles	Manager or Owner only	Generation fails, no files saved
Survey questions	3-5 per survey	Generation fails, no files saved
Response indices	Valid 0-9 (users), 0-4 (surveys)	Generation fails, no files saved

Why strict validation:

• Prevents bad data: LLMs can hallucinate or miscount
• Clear feedback: Developer knows immediately if generation failed
• No partial writes: Either all files are valid, or none are saved
• Idempotent: Safe to retry generation without cleanup

Example validation failure:

   GENERATION FAILED: Expected EXACTLY 10 users, got 9.
   This is a hard constraint. Generation must be retried.
   No files were saved. Please run generation again.

Dynamic ID Mapping

During seeding, user and survey IDs are captured from API responses and stored in memory:

# Create user, capture dynamic ID
user = client.create_user({"name": "Alice", "email": "alice@example.com"})
user_id = user["id"]  # e.g., "clx8k2j4f0000..."

# Later: Use captured ID for response submission
client.submit_response(survey_id, {"userId": user_id, ...})

Why dynamic mapping:

• No hardcoded IDs: Works regardless of Formbricks' ID generation strategy
• Order-independent: User/survey creation order doesn't matter
• API-driven: IDs are determined by Formbricks, not assumed by the client

Tradeoff: IDs are not persisted. If seeding fails midway, you must regenerate data or manually track IDs.

LLM Provider Flexibility

The system supports both OpenAI (cloud) and Ollama (local):

Provider	Pros	Cons	When to use
OpenAI	High quality, fast	Requires API key, costs money	Production, reliable results
Ollama	Free, local, private	Slower, may need prompt tuning	Development, offline work

How to switch:

# Use OpenAI (default)
LLM_PROVIDER=openai
OPENAI_API_KEY=sk-...

# Use Ollama (local)
LLM_PROVIDER=ollama
OLLAMA_BASE_URL=http://localhost:11434

The prompt explicitly instructs the LLM to return only JSON with no additional text. OpenAI's json_object mode and Ollama's json format ensure clean output.

Error Handling

Error Type	Behavior	Resolution
API 4xx/5xx	Raise exception with status and response	Check Formbricks logs, verify API key
Invalid JSON from LLM	Raise JSONDecodeError	Retry generation, check LLM provider
Validation failure	Raise ValueError, no files saved	Retry generation
Missing data files	Raise FileNotFoundError	Run generate before seed
Role assignment failure	Stop seeding, raise exception	Check Management API key, restart environment

No silent failures: All errors are raised loudly. Partial data may exist in Formbricks after a failed seed.

Tradeoffs & Future Improvements

Current Approach	Tradeoff	Potential Improvement
No rollback on seed failure	Manual cleanup required	Implement transactional seeding or cleanup script
In-memory ID mapping	Lost if process crashes	Persist mappings to generated_data/mappings.json
No retry logic	Single API failure stops everything	Add exponential backoff for transient errors
Docker via subprocess	Less robust than Docker SDK	Use docker-py for programmatic control
No progress bars	Unclear how long operations take	Add tqdm for visual progress
No parallelization	Seeding is sequential	Batch user creation or use async API calls

Example Workflow

# Start Formbricks locally
python main.py formbricks up

# Generate seed data with OpenAI
python main.py formbricks generate

# Seed the generated data into Formbricks
python main.py formbricks seed

# Stop the environment when done
python main.py formbricks down

Debugging Tips

Seeding fails with "401 Unauthorized":

• Verify FORMBRICKS_MANAGEMENT_API_KEY is set correctly in .env
• Check that you're using the Management API key, not a Client API key

LLM generates wrong counts:

• Try regenerating with python main.py formbricks generate
• If using Ollama, switch to OpenAI for more reliable results
• Check that your LLM model supports JSON mode

Partial data after failed seed:

• Run python main.py formbricks down to stop containers
• Run python main.py formbricks up to start fresh
• Regenerate and re-seed data

Role assignment fails:

• Ensure the Management API key has permission to assign roles
• Check Formbricks logs: docker logs formbricks-formbricks-1
• Verify roles are exactly "Manager" or "Owner" (case-sensitive)

Summary

This CLI system demonstrates:

• API-first architecture with proper separation between Management and Client APIs
• Explicit role assignment with mandatory validation
• Strict LLM validation to prevent bad data from entering the system
• Dynamic ID mapping for flexible, API-driven workflows
• Clear error handling with no silent failures

Notes on Local Setup

• Formbricks is orchestrated locally via Docker Compose using the official image.
• The focus of this implementation is on CLI-driven lifecycle management and API-only data generation and seeding, as specified in the challenge.
• In some environments, the initial /setup UI may return a 500 error due to upstream NextAuth / encryption secret validation behavior. This does not affect the CLI commands or the Management and Client API interactions demonstrated here.