[WIP, has hash changes] Vocal phrase refactor with supporting parser improvements - #88
Closed
elicwhite wants to merge 6 commits into
Closed
[WIP, has hash changes] Vocal phrase refactor with supporting parser improvements#88elicwhite wants to merge 6 commits into
elicwhite wants to merge 6 commits into
Conversation
elicwhite
force-pushed
the
parser-improvements
branch
8 times, most recently
from
April 17, 2026 06:33
8394b43 to
6116e21
Compare
elicwhite
changed the title
elicwhite
force-pushed
the
parser-improvements
branch
7 times, most recently
from
April 18, 2026 03:42
7fb7ddb to
29943f5
Compare
- unrecognizedEvents on RawChartData: text events from EVENTS track not already extracted into sections/endEvents/vocalTracks/coda (crowd events, music_start/end, practice mode, time signatures as text, etc.) - Read MIDI sections, end events, coda, and unrecognized events from all text-like event types (text, lyrics, marker, cuePoint), matching YARG.Core's MoonText behavior - Coda events are surfaced both in `codaEvents` AND `unrecognizedEvents` for consumer visibility (mirrors YARG/MoonSong behavior where coda appears in globalEvents) - New parse issues for stray vocal events on the EVENTS track. Game engines silently drop these; we record them so consumers can surface the misplacement: - `invalidLyric` — lyric text on EVENTS instead of PART VOCALS - `invalidPhraseStart` — phrase_start on EVENTS (vocal phrases use MIDI 105/106 on PART VOCALS) - `invalidPhraseEnd` — phrase_end on EVENTS Validated: 0 hash regressions across 78,046 charts.
…t Real Drums New instruments: proguitar, proguitar22, probass, probass22, prokeys, elitedrums, keysghl. New instrument types: proGuitar, proKeys, eliteDrums. New game modes: proGuitar, proKeys, eliteDrums. MIDI tracks recognized: - PART REAL_GUITAR, PART REAL_GUITAR_22, PART REAL_BASS, PART REAL_BASS_22 - PART REAL_KEYS_X/H/M/E (one track per difficulty, CopyDown from expert) - PART ELITE_DRUMS - PART REAL_DRUMS_PS (parsed as standard drums fallback) - PART KEYS GHL (6-fret keys, same layout as other GHL instruments) Instrument-wide data extracted (star power, solo via 116/115, text events, versus phrases, animations, Pro Keys range shifts). Per-difficulty raw notes stored in rawNotes with noteNumber/velocity/channel. Pro instrument parsing (raw notes + Pro Keys range shifts) is integrated into the existing single-pass scanInstrumentTrack flow — no separate extractors. \`storesRawNotes(instrumentType)\` and \`getRawNoteDifficulty()\` replace the "isNewInstrument" flag with meaningful, instrumentType-derived checks. PART REAL_DRUMS_PS / PART DRUMS dedup ------------------------------------- Matches YARG/Moonscraper behavior: PART DRUMS is canonical, PART REAL_DRUMS_PS is a fallback. When both exist, the fallback is dropped during parsing (YARG TrackOverrides: DRUMS_REAL_TRACK has overwrite=false). When only the fallback is present, it is parsed as drums. - New \`duplicateDrumsTrack\` ChartIssueType, emitted into the parseIssues array (added in the prior PR) when the dedup triggers - 19 charts in the 78,046-chart corpus have both PART DRUMS and PART REAL_DRUMS_PS; all now hash deterministically regardless of MIDI track order Validated: 78,046 charts, 0 hash diffs vs fresh master baseline.
…e, glissando Semantic labels on rawNotes: - Pro Guitar/Bass: string (0-5), fret (0-17/22, capped), noteModifier (normal/muted/ghost/bend/tapped/harmonics/pinchHarmonics from MIDI channel) - Pro Keys: key (0-24) - Elite Drums: pad name (hatPedal/kick/snare/hiHat/leftCrash/tom1-3/ride/rightCrash) - Sustain trimming: resolution/3 with strict < (matching YARG Pro behavior) Typed animations (name field on animations array): - Drum: 28 named pad hits (kick, snareLHHard, hiHatRight, etc.) - Guitar/Bass/Keys: leftHandPosition1-20 New trackData fields: - handMaps: typed HandMap events (10 types from "map HandMap_*" text events) - strumMaps: typed StrumMap events (3 types from "map StrumMap_*" text events) - characterStates: typed character states (7 types from [idle]/[play]/etc.) - glissandoSections: Pro Keys glissando (note 126, distinct from flexLane) Coda events now included in globalEvents (not filtered). Validated: 78,046 charts 0 hash diffs.
Parse the complete VENUE MIDI track into typed events: - Lighting: keyframes (notes 48-50) + presets (24 text patterns via "lighting (TYPE)") - PostProcessing: notes 96-110 (15 effects) + text .pp patterns (30 effects) - Camera Cuts: notes 60-64 + directed text (30+ patterns) + coop text (35+ patterns) - Camera Constraints: notes 70-73 (no_behind/only_far/only_close/no_close) - Spotlights: notes 37-41 (bass/drums/guitar/vocals/keys) - Singalongs: notes 85-87 (bass/drums/guitar) - Stage Effects: text events (bonus_fx, fog_on/off) New types: VenueEvent, VenueEventType. New field: venue[] on RawChartData/ParsedChart. Validated: 78,046 charts 0 hash diffs. Venue matches YARG for all charts with venue data (0 diffs).
scanIni() now returns unknownIniValues: a Record<string, string> of all key-value pairs from the [Song] section that aren't in the recognized property set (36 known keys + 4 legacy aliases). This enables writers to reconstruct the full song.ini without data loss. Also exports scanIni() from the package entry point. Found 206 unique unrecognized keys across 78K charts, including: - Pro instrument diffs (diff_vocals_harm, diff_guitar_real, etc.) - Phase Shift SysEx flags (sysex_slider, sysex_open_bass, etc.) - Tuning/type metadata (real_guitar_tuning, kit_type, etc.) - Various typos and custom metadata Validated: 78,046 charts 0 hash diffs.
Core change: eliminate top-level notes[]/lyrics[] from NormalizedVocalPart. All vocal content is now accessed through phrase grouping. Vocal phrase refactor: - NormalizedVocalPart: removed notes[] and lyrics[] - NormalizedVocalPhrase: added player?: 1 | 2 for versus mode - NormalizedLyricEvent.text: stores original unstripped markup symbols - NormalizedVocalNote.pitch: nonPitched keeps original MIDI pitch - PART VOCALS merges note 105+106 phrases with player tags - Harmonies keep separate 105/106 for lossless round-trip - Pre-collect lyrics per phrase (lyricIdx advances to phraseEnd regardless of notes, preventing divergence on pitch slide round-trip) - Pitch slide notes only skipped when lyric survives emptiness filter - HARM3 CopyDown: clone staticLyricPhrases from HARM2 - Writer emits union of notes/lyrics from both phrase sets Supporting changes (same files, interleaved): - Beat track: extract MIDI BEAT track for round-trip fidelity - Chart parser: resolveChartTrackName, parseChartSectionEventText - orangeDrum noteType for 5-lane drums, _endSeq modifier ordering - extractMidiVocalTextEvents for stance/facial anim events - VenueEvent: 'unknown' type and length field - New tests: drum pads (4/5-lane) and guitar force modifiers Validated: 0 vocal diffs across 15,525 charts on round-trip. Hash validation: 13,338 matched, 0 mismatched. 46/46 unit tests pass.
elicwhite
force-pushed
the
parser-improvements
branch
from
April 18, 2026 03:53
29943f5 to
78077ae
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Core change: Eliminate top-level
notes[]andlyrics[]fromNormalizedVocalPart. All vocal content is now accessed through phrase grouping (notePhrases/staticLyricPhrases).Vocal phrase refactor
NormalizedVocalPart: removednotes[]andlyrics[]NormalizedVocalPhrase: addedplayer?: 1 | 2for PART VOCALS versus modeNormalizedLyricEvent.text: stores original unstripped markup symbolsNormalizedVocalNote.pitch: nonPitched notes keep original MIDI pitch (not -1)Supporting changes (same files, interleaved)
resolveChartTrackName,parseChartSectionEventTextorangeDrumnoteType for 5-lane drums,_endSeqmodifier orderingextractMidiVocalTextEventsfor stance/facial anim eventsVenueEvent: 'unknown' type and length fieldValidation