Skip to content

Add wildcard events, template generation, count(), and array itemTransform#10

Merged
vgpastor merged 7 commits intomainfrom
claude/fix-events-templates-Bts2F
Feb 16, 2026
Merged

Add wildcard events, template generation, count(), and array itemTransform#10
vgpastor merged 7 commits intomainfrom
claude/fix-events-templates-Bts2F

Conversation

@vgpastor
Copy link
Copy Markdown
Owner

Summary

This release adds four user-requested features to improve the bulk import API: wildcard event subscription for monitoring all events at once, CSV template generation with synthetic example data, a count() method to get total records before processing, and per-item transformation for array fields.

Key Changes

  • Wildcard event subscription (onAny() / offAny())

    • Subscribe to all domain events regardless of type via importer.onAny(handler)
    • Simplifies SSE/WebSocket relay patterns where all events need forwarding
    • Handlers are isolated; errors don't propagate to other handlers
  • Enhanced generateTemplate() with example rows

    • Now accepts { exampleRows: N } option to generate synthetic data rows
    • Each field type produces appropriate examples: fake emails, ISO dates, numbers, booleans, arrays
    • Fields with defaultValue use that value instead of generated examples
    • Backward compatible: calling without options returns header-only (existing behavior)
  • New count() method

    • Returns total record count from configured source without modifying import state
    • Useful for progress bars and UI feedback before calling start()
    • Requires from() to be called first; throws if source not configured
  • Array field itemTransform option

    • New optional itemTransform: (item: string) => string on array fields
    • Applied to each element after splitting by separator
    • Enables normalization (e.g., lowercase, trim) of array items
    • Backward compatible: existing code without itemTransform works unchanged
  • Deferred import:started event emission

    • Event now fires after yielding to microtask queue (await Promise.resolve())
    • Allows handlers registered after start() on the same tick to receive the event
    • Fixes race condition where late-registered handlers would miss the event
  • API refinements

    • getStatus() now returns both status and state (deprecated alias) for backward compatibility
    • Exported ParsedRecord type for processor callback signatures
    • Exported GenerateTemplateOptions type for template generation
    • Updated RecordProcessorFn to accept ParsedRecord instead of RawRecord

Implementation Details

  • EventBus extended with wildcardHandlers Set and onAny()/offAny() methods
  • Template generation uses a private generateExampleValue() helper that switches on field type
  • count() streams through source and parser without creating import state
  • Array transformation in SchemaValidator applies itemTransform after split and trim
  • Deferred emission in StartImport uses await Promise.resolve() before emitting import:started

Testing

  • 477 lines of comprehensive acceptance tests covering all new features
  • Unit tests for EventBus wildcard handlers and error isolation
  • All existing tests updated to use status instead of deprecated state

https://claude.ai/code/session_0152RPJq1WLJhnyLDFZxQwF7

…nt, naming, itemTransform

Implements all 7 items from user feedback after integrating v0.4.0:

1. Deferred import:started — emits on next microtask so handlers registered
   after start() on the same tick receive the event.
2. generateTemplate(schema, { exampleRows }) — generates synthetic data rows
   based on field type (email, date, number, boolean, array, custom).
3. onAny()/offAny() wildcard event subscription — receive all domain events
   without listing each type. Simplifies SSE/WebSocket relay.
4. ParsedRecord type — semantic alias for RawRecord, used in RecordProcessorFn
   to indicate transforms have been applied. Non-breaking (structural typing).
5. count() method — streams source to count records without modifying state.
   Useful for progress bar initialization before start().
6. getStatus() returns status (+ deprecated state alias) for naming consistency.
7. itemTransform on FieldDefinition — per-element transform for array fields
   after splitting (e.g. s => s.toLowerCase()).

300 tests passing (was 276). Full pipeline green: typecheck, lint, test, build.

https://claude.ai/code/session_0152RPJq1WLJhnyLDFZxQwF7
8 scenarios covering full import lifecycle at realistic scale:
1. Happy path: 1500 records, 8 batches, progress + event verification
2. Mixed errors: 1500 records, every 10th invalid, continueOnError
3. Concurrency: 1200 records with maxConcurrentBatches=4
4. onAny() relay: 1000 records, full event stream order validation
5. count() consistency: verify count matches processed total
6. Combined: 2000 records + errors + concurrency + progress
7. Deferred import:started at scale
8. itemTransform on array fields across 1000 records

308 total tests (was 300).

https://claude.ai/code/session_0152RPJq1WLJhnyLDFZxQwF7
…L/MariaDB

MySQL/MariaDB drivers can return DataTypes.JSON columns as raw strings
instead of parsed objects. Added defensive parseJson() utility at all
read boundaries (updateBatchState, toDomain mappers, getProgress) to
handle both cases. Widened BIGINT column types to accept string values
from SQLite. Added 13 new tests covering JSON-as-string scenarios.

https://claude.ai/code/session_0152RPJq1WLJhnyLDFZxQwF7
Two acceptance test files had Prettier formatting issues. Updated
CLAUDE.md mandatory checklist to include `npm run format:check` so
formatting is always verified alongside typecheck, lint, test, and build.

https://claude.ai/code/session_0152RPJq1WLJhnyLDFZxQwF7
@vgpastor vgpastor merged commit f6d657f into main Feb 16, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants