# Analytics & Telemetry System (v0.5.0)

## Overview

nChat v0.5.0 introduces a comprehensive analytics and telemetry system for tracking usage, performance, and engagement metrics. Built on **TimescaleDB** for efficient time-series data storage and querying.

## Table of Contents

1. [Architecture](#architecture)
2. [Features](#features)
3. [Database Schema](#database-schema)
4. [Configuration](#configuration)
5. [Usage](#usage)
6. [API Endpoints](#api-endpoints)
7. [Dashboard](#dashboard)
8. [Privacy & Compliance](#privacy--compliance)
9. [Performance](#performance)
10. [Troubleshooting](#troubleshooting)

---

## Architecture

### Components

```
┌─────────────────┐
│  React Client   │ ──track()──▶ Analytics Client
└─────────────────┘                     │
                                        │ batch
                                        ▼
                            ┌────────────────────┐
                            │  Analytics API     │
                            │  /api/analytics/*  │
                            └────────────────────┘
                                        │
                                        ▼
                            ┌────────────────────┐
                            │  PostgreSQL +      │
                            │  TimescaleDB       │
                            │  (Hypertables)     │
                            └────────────────────┘
                                        │
                                        ▼
                            ┌────────────────────┐
                            │  Hasura GraphQL    │
                            │  (Queries/Subs)    │
                            └────────────────────┘
                                        │
                                        ▼
                            ┌────────────────────┐
                            │  Analytics         │
                            │  Dashboard         │
                            └────────────────────┘
```

### Technology Stack

- **Database**: PostgreSQL 15+ with TimescaleDB extension
- **Time-Series Storage**: TimescaleDB hypertables with automatic partitioning
- **API**: Next.js API routes
- **GraphQL**: Hasura for real-time queries and subscriptions
- **Client**: React hooks + Zustand for state management
- **Visualization**: Recharts (lightweight, customizable)
- **Export**: CSV, JSON, PDF support

---

## Features

### 1. Usage Analytics

- **User Activity**
  - Daily/Weekly/Monthly Active Users (DAU/WAU/MAU)
  - User retention cohorts
  - Session duration tracking
  - Feature adoption rates

- **Message Analytics**
  - Message volume by channel/user/time
  - Average message length
  - Attachments vs text-only
  - Thread participation rates

- **Channel Analytics**
  - Channel growth over time
  - Active vs inactive channels
  - Engagement rates by channel
  - Member participation

### 2. Performance Analytics

- **API Performance**
  - Response times (p50/p95/p99)
  - Error rates by endpoint
  - Request volume by endpoint
  - Database query performance

- **WebSocket Metrics**
  - Connection health
  - Message latency
  - Packet loss rates
  - Concurrent connections

### 3. Engagement Metrics

- **Reaction Analytics**
  - Most used reactions
  - Reaction diversity
  - Top reactors

- **File Analytics**
  - Upload volume and size
  - File type breakdown
  - Storage usage per user

- **Search Analytics**
  - Popular search queries
  - No-results searches
  - Click-through rates

### 4. Business Metrics

- User growth rate
- Storage usage trends
- Feature usage by tier
- API call volume

---

## Database Schema

### Core Tables

#### 1. `nchat_analytics_events` (Hypertable)

Primary time-series table for all analytics events.

```sql
CREATE TABLE nchat_analytics_events (
  id UUID PRIMARY KEY,
  timestamp TIMESTAMPTZ NOT NULL,
  event_name VARCHAR(100) NOT NULL,
  event_category VARCHAR(50) NOT NULL,
  user_id UUID REFERENCES nchat_users(id),
  session_id VARCHAR(100) NOT NULL,
  platform VARCHAR(20),
  properties JSONB DEFAULT '{}'
);

-- Convert to hypertable (1 day chunks)
SELECT create_hypertable('nchat_analytics_events', 'timestamp',
  chunk_time_interval => INTERVAL '1 day');

-- Retention: 90 days
SELECT add_retention_policy('nchat_analytics_events', INTERVAL '90 days');
```

#### 2. `nchat_analytics_user_activity` (Hypertable)

Aggregated hourly user activity metrics.

```sql
CREATE TABLE nchat_analytics_user_activity (
  timestamp TIMESTAMPTZ NOT NULL,
  user_id UUID NOT NULL,
  messages_sent INTEGER DEFAULT 0,
  reactions_given INTEGER DEFAULT 0,
  files_uploaded INTEGER DEFAULT 0,
  engagement_score FLOAT DEFAULT 0,
  session_duration_seconds INTEGER DEFAULT 0,
  UNIQUE(user_id, timestamp)
);

SELECT create_hypertable('nchat_analytics_user_activity', 'timestamp',
  chunk_time_interval => INTERVAL '7 days');
```

#### 3. `nchat_analytics_channel_activity` (Hypertable)

Aggregated hourly channel activity metrics.

```sql
CREATE TABLE nchat_analytics_channel_activity (
  timestamp TIMESTAMPTZ NOT NULL,
  channel_id UUID NOT NULL,
  messages_count INTEGER DEFAULT 0,
  active_users_count INTEGER DEFAULT 0,
  reactions_count INTEGER DEFAULT 0,
  average_response_time_seconds FLOAT DEFAULT 0,
  UNIQUE(channel_id, timestamp)
);
```

#### 4. Continuous Aggregates (Materialized Views)

Automatically maintained aggregated views for fast queries:

```sql
-- Daily user summary
CREATE MATERIALIZED VIEW nchat_analytics_daily_user_summary
WITH (timescaledb.continuous) AS
SELECT
  time_bucket('1 day', timestamp) AS day,
  user_id,
  SUM(messages_sent) AS total_messages,
  AVG(engagement_score) AS avg_engagement_score
FROM nchat_analytics_user_activity
GROUP BY day, user_id;

-- Refresh every hour
SELECT add_continuous_aggregate_policy(
  'nchat_analytics_daily_user_summary',
  start_offset => INTERVAL '7 days',
  end_offset => INTERVAL '1 hour',
  schedule_interval => INTERVAL '1 hour'
);
```

### Helper Functions

```sql
-- Get Daily Active Users
CREATE FUNCTION nchat_analytics_get_dau(start_date DATE, end_date DATE)
RETURNS TABLE (date DATE, active_users BIGINT);

-- Get Monthly Active Users
CREATE FUNCTION nchat_analytics_get_mau(start_date DATE)
RETURNS BIGINT;

-- Calculate Engagement Score
CREATE FUNCTION nchat_analytics_calculate_engagement_score(
  p_messages INT, p_reactions INT, p_files INT, p_threads INT
)
RETURNS FLOAT;
```

---

## Configuration

### AppConfig

Add to `src/config/app-config.ts`:

```typescript
analytics: {
  enabled: boolean // Master toggle
  trackingMode: 'full' | 'privacy' | 'minimal'
  retentionDays: number // Data retention (default: 90)
  enableRealtime: boolean // Real-time dashboard updates
  enableExports: boolean // Allow CSV/PDF exports
  anonymizeUserData: boolean // Anonymize PII
  excludeTestUsers: boolean // Exclude test users from stats
  trackPerformance: boolean // Track API performance
  trackWebSocket: boolean // Track WebSocket metrics
  trackFeatureUsage: boolean // Track feature adoption
  trackErrors: boolean // Track errors (Sentry integration)
  allowUserOptOut: boolean // Allow users to opt-out
}
```

### Environment Variables

```bash
# Required
NEXT_PUBLIC_ANALYTICS_ENABLED=true

# Optional
ANALYTICS_RETENTION_DAYS=90
TIMESCALEDB_ENABLED=true
ANALYTICS_BATCH_SIZE=10
ANALYTICS_FLUSH_INTERVAL=30000
```

---

## Usage

### Client-Side Tracking

#### Basic Event Tracking

```typescript
import { useAnalytics } from '@/hooks/use-analytics';

function MyComponent() {
  const { track } = useAnalytics();

  const handleClick = () => {
    track('button_clicked', {
      buttonName: 'Submit',
      location: 'chat-input',
    });
  };

  return <button onClick={handleClick}>Submit</button>;
}
```

#### User Identification

```typescript
import { useAnalytics } from '@/hooks/use-analytics'

function AuthComponent() {
  const { identify } = useAnalytics()

  useEffect(() => {
    if (user) {
      identify(user.id, {
        email: user.email,
        displayName: user.displayName,
        role: user.role,
        plan: 'free',
      })
    }
  }, [user])
}
```

#### Page View Tracking

```typescript
import { useAnalytics } from '@/hooks/use-analytics'

function Page() {
  const { page } = useAnalytics()

  useEffect(() => {
    page('/chat', 'Chat - nChat')
  }, [])
}
```

#### Convenience Methods

```typescript
const {
  trackMessageSent,
  trackMessageEdited,
  trackReactionAdded,
  trackFileUploaded,
  trackSearchPerformed,
  trackFeatureUsed,
  trackError,
} = useAnalytics()

// Track message sent
trackMessageSent(channelId, messageLength, hasAttachments)

// Track reaction
trackReactionAdded(messageId, '👍')

// Track file upload
trackFileUploaded(fileId, fileName, fileSize, mimeType)

// Track search
trackSearchPerformed(query, resultCount)

// Track feature usage
trackFeatureUsed('voice-call', { duration: 120 })

// Track error
trackError('api_error', 'Failed to send message', { code: 500 })
```

### Dashboard Consumption

#### useAnalyticsDashboard Hook

```typescript
import { useAnalyticsDashboard } from '@/hooks/use-analytics-dashboard';

function AnalyticsDashboard() {
  const {
    summary,
    messageVolume,
    userActivity,
    isLoading,
    error,
    setDateRange,
    refresh,
  } = useAnalyticsDashboard();

  if (isLoading) return <Spinner />;
  if (error) return <Error message={error} />;

  return (
    <div>
      <h1>Analytics Dashboard</h1>
      <MetricCard value={summary.messages.total.value} label="Total Messages" />
      <MetricCard value={summary.users.activeUsers.value} label="Active Users" />
      <Chart data={messageVolume} />
    </div>
  );
}
```

#### Specialized Hooks

```typescript
// Message analytics
const { volume, topMessages, stats } = useMessageAnalytics()

// User analytics
const { activity, growth, activeUsers } = useUserAnalytics()

// Channel analytics
const { activity, stats } = useChannelAnalytics()

// Date range filtering
const { dateRange, setPreset, setRange } = useDateRangeFilter()
setPreset('last30days')
setRange(new Date('2025-01-01'), new Date('2025-01-31'))

// Real-time updates (auto-refresh every 30 seconds)
useRealtimeAnalytics(30000)
```

---

## API Endpoints

### 1. Track Events

```
POST /api/analytics/track
Content-Type: application/json

[
  {
    "name": "message_sent",
    "category": "messaging",
    "properties": {
      "channelId": "uuid",
      "messageLength": 42
    }
  }
]

Response: { success: true, tracked: 1 }
```

### 2. Get Dashboard Data

```
GET /api/analytics/dashboard?start=2025-01-01&end=2025-01-31&granularity=day

Response: {
  success: true,
  data: {
    summary: { ... },
    messageVolume: [...],
    userActivity: [...],
    ...
  }
}
```

### 3. Get Section Data

```
GET /api/analytics/messages
GET /api/analytics/users
GET /api/analytics/channels
GET /api/analytics/reactions
```

### 4. Export Data

```
POST /api/analytics/export
Content-Type: application/json

{
  "format": "csv",
  "sections": ["summary", "messages", "users"],
  "dateRange": { "start": "2025-01-01", "end": "2025-01-31" }
}

Response: {
  success: true,
  downloadUrl: "/api/analytics/download/export-123.csv"
}
```

---

## Dashboard

### Access

Navigate to `/admin/analytics` (admin/owner only).

### Views

1. **Overview** - High-level metrics and trends
2. **Messages** - Message volume, top messages, patterns
3. **Users** - User activity, growth, retention
4. **Channels** - Channel engagement, top channels
5. **Reactions** - Reaction analytics, top emojis
6. **Files** - File uploads, storage usage
7. **Search** - Search queries, no-results searches
8. **Reports** - Scheduled reports, export history

### Filters

- **Date Range**: Today, Yesterday, Last 7/30/90 days, Custom
- **Granularity**: Hour, Day, Week, Month, Year
- **Channels**: Filter by specific channels
- **Users**: Filter by specific users
- **Include Bots**: Toggle bot inclusion

### Exports

Supported formats:

- **CSV**: Spreadsheet-friendly
- **JSON**: API-friendly
- **PDF**: Report-ready (with charts)
- **XLSX**: Excel-compatible

---

## Privacy & Compliance

### GDPR Compliance

- **Data Minimization**: Only collect necessary data
- **Anonymization**: Option to anonymize user data
- **Right to Access**: Users can export their data
- **Right to Erasure**: Users can request deletion
- **Consent Management**: Opt-in/opt-out support

### Data Retention

- **Raw Events**: 90 days (configurable)
- **Aggregated Data**: 365 days
- **Continuous Aggregates**: Permanent (summary only)
- **Automatic Cleanup**: TimescaleDB retention policies

### User Opt-Out

```typescript
// Allow users to opt-out
const { reset } = useAnalytics()
reset() // Clears user identity and stops tracking
```

---

## Performance

### Optimizations

1. **Client-Side Batching**: Events batched and sent every 30 seconds
2. **Server-Side Caching**: Dashboard data cached for 5 minutes
3. **Continuous Aggregates**: Pre-computed summaries for fast queries
4. **TimescaleDB Compression**: Automatic compression of old data
5. **Index Optimization**: Strategic indexes on high-traffic queries

### Query Performance

- DAU/MAU queries: <100ms
- Dashboard load: <500ms
- Event insertion: <10ms (batched)
- Export generation: <5s for 30 days

### Scalability

- **10M+ events/day**: Tested and optimized
- **100k+ concurrent users**: Supported
- **10TB+ data**: TimescaleDB handles efficiently

---

## Troubleshooting

### Issue: Events Not Tracking

**Solution:**

1. Check `analytics.enabled` in AppConfig
2. Verify `NEXT_PUBLIC_ANALYTICS_ENABLED=true`
3. Check browser console for errors
4. Verify analytics client initialized

### Issue: Dashboard Slow

**Solution:**

1. Reduce date range
2. Increase cache TTL
3. Check TimescaleDB compression status
4. Verify continuous aggregates are refreshing

### Issue: High Database Load

**Solution:**

1. Enable continuous aggregates
2. Increase retention policy frequency
3. Check for missing indexes
4. Enable TimescaleDB compression

### Issue: Missing Data

**Solution:**

1. Check retention policies (data may be expired)
2. Verify hypertables are created
3. Check Hasura permissions
4. Verify cron jobs running (continuous aggregates)

---

## Migration Guide

### From v0.4.0 to v0.5.0

1. **Enable TimescaleDB**:

   ```bash
   cd .backend
   nself db exec postgres "CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE;"
   ```

2. **Run Analytics Migration**:

   ```bash
   nself db migrate up
   # Applies analytics schema from migrations/
   ```

3. **Update AppConfig**:

   ```typescript
   analytics: {
     enabled: true,
     trackingMode: 'full',
     retentionDays: 90,
     // ...
   }
   ```

4. **Initialize Analytics Client**:

   ```typescript
   // Add to _app.tsx or root layout
   import { getAnalyticsClient } from '@/lib/analytics/analytics-client'

   useEffect(() => {
     getAnalyticsClient().initialize()
   }, [])
   ```

---

## Best Practices

1. **Track Meaningful Events**: Don't over-track; focus on business-critical events
2. **Use Batching**: Batch events for better performance
3. **Respect Privacy**: Anonymize PII, honor opt-outs
4. **Monitor Performance**: Watch for slow queries
5. **Regular Exports**: Export data regularly for backup
6. **Dashboard Access**: Restrict to admins only
7. **Test Tracking**: Use test users excluded from stats

---

## Roadmap

### v0.5.1

- [ ] Enhanced PDF exports with charts
- [ ] Email scheduled reports
- [ ] Custom dashboard widgets

### v0.6.0

- [ ] Machine learning insights
- [ ] Anomaly detection
- [ ] Predictive analytics

---

## Support

For issues or questions:

- GitHub Issues: https://github.com/nself/nchat/issues
- Documentation: https://docs.nchat.io/analytics
- Email: support@nself.org

---

**Version**: v0.5.0
**Last Updated**: 2026-01-30
**Contributors**: nself team