Add wildcard events, template generation, count(), and array itemTransform#10
Merged
Add wildcard events, template generation, count(), and array itemTransform#10
Conversation
…nt, naming, itemTransform
Implements all 7 items from user feedback after integrating v0.4.0:
1. Deferred import:started — emits on next microtask so handlers registered
after start() on the same tick receive the event.
2. generateTemplate(schema, { exampleRows }) — generates synthetic data rows
based on field type (email, date, number, boolean, array, custom).
3. onAny()/offAny() wildcard event subscription — receive all domain events
without listing each type. Simplifies SSE/WebSocket relay.
4. ParsedRecord type — semantic alias for RawRecord, used in RecordProcessorFn
to indicate transforms have been applied. Non-breaking (structural typing).
5. count() method — streams source to count records without modifying state.
Useful for progress bar initialization before start().
6. getStatus() returns status (+ deprecated state alias) for naming consistency.
7. itemTransform on FieldDefinition — per-element transform for array fields
after splitting (e.g. s => s.toLowerCase()).
300 tests passing (was 276). Full pipeline green: typecheck, lint, test, build.
https://claude.ai/code/session_0152RPJq1WLJhnyLDFZxQwF7
8 scenarios covering full import lifecycle at realistic scale: 1. Happy path: 1500 records, 8 batches, progress + event verification 2. Mixed errors: 1500 records, every 10th invalid, continueOnError 3. Concurrency: 1200 records with maxConcurrentBatches=4 4. onAny() relay: 1000 records, full event stream order validation 5. count() consistency: verify count matches processed total 6. Combined: 2000 records + errors + concurrency + progress 7. Deferred import:started at scale 8. itemTransform on array fields across 1000 records 308 total tests (was 300). https://claude.ai/code/session_0152RPJq1WLJhnyLDFZxQwF7
…L/MariaDB MySQL/MariaDB drivers can return DataTypes.JSON columns as raw strings instead of parsed objects. Added defensive parseJson() utility at all read boundaries (updateBatchState, toDomain mappers, getProgress) to handle both cases. Widened BIGINT column types to accept string values from SQLite. Added 13 new tests covering JSON-as-string scenarios. https://claude.ai/code/session_0152RPJq1WLJhnyLDFZxQwF7
Two acceptance test files had Prettier formatting issues. Updated CLAUDE.md mandatory checklist to include `npm run format:check` so formatting is always verified alongside typecheck, lint, test, and build. https://claude.ai/code/session_0152RPJq1WLJhnyLDFZxQwF7
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This release adds four user-requested features to improve the bulk import API: wildcard event subscription for monitoring all events at once, CSV template generation with synthetic example data, a
count()method to get total records before processing, and per-item transformation for array fields.Key Changes
Wildcard event subscription (
onAny()/offAny())importer.onAny(handler)Enhanced
generateTemplate()with example rows{ exampleRows: N }option to generate synthetic data rowsdefaultValueuse that value instead of generated examplesNew
count()methodstart()from()to be called first; throws if source not configuredArray field
itemTransformoptionitemTransform: (item: string) => stringon array fieldsitemTransformworks unchangedDeferred
import:startedevent emissionawait Promise.resolve())start()on the same tick to receive the eventAPI refinements
getStatus()now returns bothstatusandstate(deprecated alias) for backward compatibilityParsedRecordtype for processor callback signaturesGenerateTemplateOptionstype for template generationRecordProcessorFnto acceptParsedRecordinstead ofRawRecordImplementation Details
wildcardHandlersSet andonAny()/offAny()methodsgenerateExampleValue()helper that switches on field typecount()streams through source and parser without creating import stateitemTransformafter split and trimawait Promise.resolve()before emittingimport:startedTesting
statusinstead of deprecatedstatehttps://claude.ai/code/session_0152RPJq1WLJhnyLDFZxQwF7