feat: ETL steps for star systems and orbits#212
Merged
Conversation
- Add StarSystemsSyncStep: upserts station_star_system on uex_id; maps faction_uex_id, jurisdiction_uex_id, wiki, boolean flags, and timestamps; warns and skips records missing a name - Add OrbitsSyncStep: upserts station_orbit on uex_id after validating id_star_system against the set of known star systems in DB; emits a top-level warning when station_star_system is empty (star-systems step likely failed); warns and skips orbits with null or unknown star system Also syncs station_orbit_distance from /orbit_distances; skips and warns when either orbit FK is unknown - Register both steps in CatalogEtlModule and CatalogEtlService (star systems at tier-2, orbits at tier-3 per dependency order) - Add mock providers for StarSystemsSyncStep and OrbitsSyncStep in the CatalogEtlService spec - 15 new unit tests; all 41 catalog-etl tests passing; typecheck clean Closes #191
… ETL - Move starSystemsSyncStep before factionsSyncStep so faction→star system junction reconciliation finds populated rows on the first run - Guard faction_uex_id and jurisdiction_uex_id writes in StarSystemsSyncStep and syncOrbits() — unknown FKs are stored as null with a warning instead of raising a DB FK violation - Guard star_system_origin/dest_uex_id writes in syncOrbitDistances() — unknown star system FKs are stored as null with a warning instead of raising a DB FK violation - Add unit tests covering all three failure modes (8 new tests across 2 spec files; 32 total passing)
Move reconcileStarSystems() from FactionsSyncStep into StarSystemsSyncStep so it runs after both factions and star systems are populated. Step order is now factions → jurisdictions → starSystems → companies → orbits: - StarSystemsSyncStep re-fetches /factions at the end of execute() and reconciles station_faction_star_system in full (delete + re-insert), replicating the prior FactionsSyncStep logic but in the correct position - FactionsSyncStep no longer touches station_faction_star_system - Junction reconciliation tests moved from factions-sync.step.spec.ts to star-systems-sync.step.spec.ts; 44 unit tests passing
… factions-sync Load station_faction rows at the start of reconcileStarSystemJunctions() and skip any faction from /factions that was not persisted — warns and skips the DELETE+INSERT block entirely to avoid a FK violation on station_faction_star_system. Adds test covering a named faction absent from the DB.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/star_systems, upsertsstation_star_systemonuex_id; mapsfaction_uex_id,jurisdiction_uex_id, boolean flags, wiki, and timestamps; warns and skips records missing a name/orbits, validates each orbit'sid_star_systemagainst the currentstation_star_systemtable before inserting; emits a top-level warning if the table is empty (upstream step likely failed); warns and skips orbits with null or unknown star system FK; also fetches/orbit_distances, validates both orbit FKs, and upsertsstation_orbit_distance— warns and skips on any unknown FKCatalogEtlModuleandCatalogEtlServiceat tier-2 (star systems) and tier-3 (orbits) per dependency orderCatalogEtlServicespec updated with mock providers for the two new stepsTest plan
SELECT count(*) FROM station_star_systemmatches UEX countstation_orbitrows have a non-nullstar_system_uex_idid_star_systemappears instation_etl_warningand is absent fromstation_orbitCloses #191