new importer

IMPORT_REVIEW.md

@@ -0,0 +1,338 @@
+# Import Package Code Review
+
+## Summary
+Successfully implemented a flexible, provider-based import system for analytics data. The architecture is solid, but there are several improvements we can make.
+
+---
+
+## 🔴 Critical Issues
+
+### 1. **Type Safety in Import Job**
+**Location:** `apps/worker/src/jobs/import.ts`
+
+**Issue:** Extensive use of `any` types throughout the file
+```typescript
+const eventBatch: any[] = [];  // Line 39
+provider: any,                 // Line 113
+events: any[],                 // Line 169
+```
+
+**Fix:** Create proper types
+```typescript
+import type { BaseImportProvider } from '@openpanel/importer';
+
+async function processBatch<T = unknown>(
+  rawEvents: T[],
+  provider: BaseImportProvider<T>,
+  sessionReconstructor: SessionReconstructor,
+  jobLogger: Logger,
+  batchNumber: number
+)
+```
+
+---
+
+## 🟡 Medium Priority Issues
+
+### 2. **Missing UUID Dependency**
+**Location:** `packages/importer/package.json`
+
+**Issue:** `uuid` is used but not in dependencies
+```json
+"dependencies": {
+  "csv-parser": "^3.0.0",
+  "ramda": "^0.29.1"
+  // uuid is missing!
+}
+```
+
+**Fix:** Add to dependencies
+```json
+"dependencies": {
+  "@openpanel/db": "workspace:*",
+  "@openpanel/common": "workspace:*",
+  "@openpanel/queue": "workspace:*",
+  "csv-parser": "^3.0.0",
+  "ramda": "^0.29.1",
+  "uuid": "^9.0.1"
+}
+```
+
+### 3. **Session Reconstructor Mutates Input**
+**Location:** `packages/importer/src/session-reconstructor.ts:23`
+
+**Issue:** Directly mutates the event object
+```typescript
+event.session_id = session.id;  // Mutates input
+```
+
+**Fix:** Return new objects instead
+```typescript
+const processedEvent = {
+  ...event,
+  session_id: session.id
+};
+results.push(processedEvent);
+```
+
+### 4. **Missing Error Handling for Invalid Timestamps**
+**Location:** `packages/importer/src/session-reconstructor.ts:18,24,39`
+
+**Issue:** No validation for invalid dates
+```typescript
+new Date(event.created_at).getTime()  // Could be NaN
+```
+
+**Fix:** Add validation
+```typescript
+private getTimestamp(dateString: string | undefined): number {
+  if (!dateString) return Date.now();
+  const timestamp = new Date(dateString).getTime();
+  return Number.isNaN(timestamp) ? Date.now() : timestamp;
+}
+```
+
+### 5. **Missing Types Export**
+**Location:** `packages/importer/package.json`
+
+**Issue:** `@types/csv-parser` not in devDependencies
+```json
+"devDependencies": {
+  "@types/node": "^20.0.0",
+  "typescript": "^5.0.0",
+  "vitest": "^1.0.0"
+  // Missing @types/csv-parser
+}
+```
+
+**Fix:** Add missing type definitions
+
+### 6. **Import Job Progress Calculation**
+**Location:** `apps/worker/src/jobs/import.ts:70`
+
+**Issue:** `totalBatches` calculation is inaccurate during processing
+```typescript
+totalBatches: Math.ceil(totalEvents / BATCH_SIZE)  // Changes as totalEvents grows
+```
+
+**Fix:** Calculate total batches at the end or estimate better
+
+---
+
+## 🟢 Nice-to-Have Improvements
+
+### 7. **Configurable Session Timeout**
+**Current:** Hardcoded 30-minute timeout
+```typescript
+private readonly SESSION_TIMEOUT = 30 * 60 * 1000;
+```
+
+**Improvement:** Make it configurable per provider
+```typescript
+constructor(private readonly sessionTimeout = 30 * 60 * 1000) {}
+```
+
+### 8. **Better Logging for Skipped Events**
+**Current:** Logs entire raw event
+```typescript
+jobLogger.warn('Skipping invalid event', { rawEvent });
+```
+
+**Improvement:** Log only relevant fields to avoid noise
+```typescript
+jobLogger.warn('Skipping invalid event', { 
+  event_id: rawEvent.event_id,
+  session_id: rawEvent.session_id,
+  reason: 'validation_failed'
+});
+```
+
+### 9. **Memory Cleanup Optimization**
+**Current:** Cleanup runs on every event
+```typescript
+this.cleanupStaleSessions(currentTimestamp);  // Called for EVERY event
+```
+
+**Improvement:** Cleanup periodically (e.g., every 1000 events)
+```typescript
+if (this.processedCount % 1000 === 0) {
+  this.cleanupStaleSessions(currentTimestamp);
+}
+```
+
+### 10. **Provider Registry Pattern**
+**Current:** Switch statement for providers
+```typescript
+function createProvider(providerName: string, config: any) {
+  switch (providerName) {
+    case 'umami': return new UmamiProvider(config);
+    default: throw new Error(`Unknown provider: ${providerName}`);
+  }
+}
+```
+
+**Improvement:** Registry pattern for extensibility
+```typescript
+class ProviderRegistry {
+  private static providers = new Map<string, typeof BaseImportProvider>();
+
+  static register(name: string, provider: typeof BaseImportProvider) {
+    this.providers.set(name, provider);
+  }
+
+  static create(name: string, config: ImportConfig) {
+    const Provider = this.providers.get(name);
+    if (!Provider) throw new Error(`Unknown provider: ${name}`);
+    return new Provider(config);
+  }
+}
+
+// Usage:
+ProviderRegistry.register('umami', UmamiProvider);
+ProviderRegistry.register('mixpanel', MixpanelProvider);
+```
+
+### 11. **Database Transaction Support**
+**Current:** Events inserted individually
+```typescript
+await Promise.all(events.map(event => createEvent(event)));
+```
+
+**Improvement:** Consider batch insert optimization or transaction support
+```typescript
+await createEventsBatch(events);  // Batch insert to ClickHouse
+```
+
+### 12. **Import Metadata Tracking**
+**Issue:** Import table created in schema but never used
+
+**Improvement:** Update import status in database
+```typescript
+// At start
+await db.import.update({
+  where: { id: importId },
+  data: { status: 'processing', totalEvents: 0 }
+});
+
+// During processing
+await db.import.update({
+  where: { id: importId },
+  data: { 
+    processedEvents,
+    totalEvents,
+    status: 'processing'
+  }
+});
+
+// On completion
+await db.import.update({
+  where: { id: importId },
+  data: { 
+    status: 'completed',
+    completedAt: new Date()
+  }
+});
+```
+
+### 13. **Test Coverage**
+**Current:** Basic unit tests created
+
+**Improvement:** Add
+- Integration tests with real Redis
+- Performance tests with large datasets
+- Error recovery tests (resume failed imports)
+
+### 14. **Provider-Specific Configuration**
+**Current:** All providers share same config structure
+
+**Improvement:** Allow provider-specific options
+```typescript
+interface ImportConfig {
+  projectId: string;
+  provider: string;
+  sourceType: 'file' | 'api';
+  sourceLocation: string;
+  providerOptions?: Record<string, unknown>; // Provider-specific config
+}
+
+// Usage for Umami:
+providerOptions: {
+  dateFormat: 'ISO8601',
+  timezone: 'UTC',
+  customFieldMapping: { ... }
+}
+```
+
+### 15. **Resumable Imports**
+**Current:** No resume capability
+
+**Improvement:** Track progress and allow resume
+```typescript
+interface ImportProgress {
+  lastProcessedEventId?: string;
+  lastBatchNumber?: number;
+  checkpointTimestamp?: string;
+}
+
+// Store checkpoint after each batch
+await saveImportCheckpoint(importId, {
+  lastProcessedEventId: events[events.length - 1].id,
+  lastBatchNumber: currentBatch,
+  checkpointTimestamp: new Date().toISOString()
+});
+```
+
+---
+
+## 📊 Architecture Strengths
+
+✅ **Clean separation of concerns** - Provider, session reconstruction, and job logic well separated  
+✅ **Extensible design** - Easy to add new providers  
+✅ **Memory efficient** - Streaming approach with batching  
+✅ **Good error handling** - Provider hooks for custom error handling  
+✅ **Reuses existing buffers** - Integrates well with existing event processing  
+✅ **Comprehensive tests** - Good test coverage structure  
+
+---
+
+## 🎯 Priority Recommendations
+
+**Must Fix (Before Production):**
+1. Add `uuid` to package.json dependencies
+2. Fix type safety in import.ts
+3. Add validation for invalid timestamps
+4. Implement import metadata tracking
+
+**Should Fix (Soon):**
+5. Make session timeout configurable
+6. Improve memory cleanup efficiency
+7. Add @types/csv-parser to devDependencies
+
+**Nice to Have (Future):**
+8. Implement provider registry pattern
+9. Add resumable imports feature
+10. Optimize batch insert performance
+11. Add comprehensive integration tests
+
+---
+
+## 📝 Code Quality Metrics
+
+- **Type Safety:** 6/10 (needs improvement)
+- **Error Handling:** 8/10 (good, could be better)
+- **Testability:** 9/10 (excellent)
+- **Extensibility:** 9/10 (excellent)
+- **Performance:** 8/10 (good, streaming approach)
+- **Documentation:** 7/10 (good comments, could use more)
+
+---
+
+## 🚀 Next Steps
+
+1. Fix critical issues (uuid dependency, type safety)
+2. Add import metadata tracking to database
+3. Test with larger datasets (100k+ events)
+4. Add monitoring/observability (metrics, alerts)
+5. Document provider creation guide for future providers
+6. Consider adding import preview feature (validate without importing)
+

apps/start/package.json

@@ -3,7 +3,7 @@
   "private": true,
   "type": "module",
   "scripts": {
-    "testing": "pnpm dev",
+    "testing2": "pnpm dev",
     "dev": "pnpm with-env vite dev --port 3000",
     "start_deprecated": "pnpm with-env node .output/server/index.mjs",
     "preview": "vite preview",

apps/start/src/components/integrations/integration-card.tsx

@@ -50,12 +50,15 @@ export function IntegrationCardHeaderButtons({
 export function IntegrationCardLogoImage({
   src,
   backgroundColor,
+  className,
 }: {
   src: string;
   backgroundColor: string;
+  className?: string;
 }) {
   return (
     <IntegrationCardLogo
+      className={className}
       style={{
         backgroundColor,
       }}

apps/start/src/components/ui/badge.tsx

@@ -9,6 +9,11 @@ const badgeVariants = cva(
   {
     variants: {
       variant: {
+        success:
+          'border-transparent bg-emerald-700 text-white [a&]:hover:bg-emerald-700/90',
+        warning:
+          'border-transparent bg-yellow-500 text-white [a&]:hover:bg-yellow-500/90',
+        info: 'border-transparent bg-blue-500 text-white [a&]:hover:bg-blue-500/90',
         default:
           'border-transparent bg-primary text-primary-foreground [a&]:hover:bg-primary/90',
         secondary: