fix(data-import): Fix/testmo import fixes#230
Merged
therealbrad merged 8 commits intomainfrom Apr 22, 2026
Merged
Conversation
…rovements - Fix unique constraint violations across all import entities by removing isDeleted filter from dedup checks (constraint does not include isDeleted). Affected: RepositoryCases, Templates, Tags, Groups, Roles, MilestoneTypes, Status, Integration, CaseFields, ResultFields. Soft-deleted records are restored on match instead of creating duplicates that violate constraints. - Remove dangerous fallback lookup in automationImports that matched wrong records and caused className update collisions. - Replace try-catch-inside-transaction patterns with upsert or check-before-create to avoid poisoning PostgreSQL transactions. - Fix missing i18n keys in TestmoMappingConfigurator by reusing existing translation namespaces and adding truly new keys to mapping namespace. - Add dataset label translations for all Testmo dataset types. - Add start:workers:prod script for built worker execution. - Fix import panel reset button to always be clickable when stuck. - Add per-phase timing instrumentation to import worker. - Increase staging batch load size from 100 to 5000 rows. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use observed per-phase processing rates to estimate remaining import time instead of a single blended average. Each entity type is estimated independently and summed for total ETA. Rates from completed phases update dynamically; default rates from production benchmarks are used for phases that have not started yet. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… import Introduced a new `entityRates` property in the import context to allow for per-entity rate-based estimates of remaining import time. Default rates are defined for various entity types, enhancing the accuracy of estimated time remaining during the import process. This change improves the granularity of progress metrics by utilizing observed rates from completed phases and dynamically updating them as new phases are processed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Expanded the Spanish and French JSON translation files by adding new keys for various entities, including workflow states, templates, roles, and session-related terms. This update improves localization support and ensures consistency across the application for Spanish and French users.
- Add retry action to import job API: failed/canceled jobs can be reset to READY so users can reconfigure mappings and re-import without re-uploading or re-analyzing the file. - Add Reconfigure and retry button in import UI when job fails during import phase. Analysis data is preserved on import failure. - Update default entity rate estimates based on observed v2 import performance (with batch size fix). testRunCases: 30 -> 725 items/sec, users: 600 -> 19, and other phases adjusted to actuals. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
# Conflicts: # testplanit/dist/workers/testmoImportWorker.js.map
Contributor
Author
|
🎉 This PR is included in version 0.22.5 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fix Testmo import failures on large exports and improve import performance, progress reporting, and error recovery.
isDeleted)25P02start:workers:prodscript for running built workersRelated Issue
N/A
Type of Change
How Has This Been Tested?
Test Configuration:
Successfully imported a 12GB Testmo export (3.4M records, 98 datasets) with zero errors. Import completed in 5h 40m (down from 7h 30m — 25% improvement). Tested against both a cloned production database and a clean seeded database. Verified retry flow: failed import → reconfigure mapping → re-import without re-upload.
Checklist
Screenshots (if applicable)
N/A
Additional Notes
The two slowest import phases remain
testRunStepResults(~42 items/sec) andrepositoryCases(~54 items/sec). These are bottlenecked by per-row Prismacreate()calls and would benefit from bulk insert optimization in the ZenStack v3 migration (Kysely).