Implementing auto-generated timelineReplace for Oopsyraidsy

[Jaehyuk-Lee]

This is a follow-up issue after #947.

Feature(s) Requested

gen_oopsy_timeline_replace.ts automatically generates Oopsyraidsy's timelineReplace.

Draft: https://github.com/Jaehyuk-Lee/cactbot/blob/oopsy-i18n-data/util/gen_oopsy_timeline_replace.ts

See auto-generated examples

See: Jaehyuk-Lee@fd7ffad

CLI Output

$ node --loader=ts-node/esm util/gen_oopsy_timeline_replace.ts
(node:34236) ExperimentalWarning: `--experimental-loader` may be removed in the future; instead use `register()`:
--import 'data:text/javascript,import { register } from "node:module"; import { pathToFileURL } from "node:url"; register("ts-node/esm", pathToFileURL("./"));'
(Use `node --trace-warnings ...` to show where the warning was created)
(node:34236) [DEP0180] DeprecationWarning: fs.Stats constructor is deprecated.
(Use `node --trace-deprecation ...` to show where the warning was created)
Alert: ui/oopsyraidsy/data/05-shb/raid/e12s.ts: Multiple candidates for 'Ice Pillar' in cn: 冰柱, 氷柱
Alert:          Using existing translation: '冰柱'
Alert: ui/oopsyraidsy/data/05-shb/raid/e12s.ts: Multiple candidates for 'Ice Pillar' in tc: 冰柱, 氷柱
Alert:          Using existing translation: '冰柱'
Success: Updated: 13, Skipped: 242

How this work

1. Search and process all data files from ui/oopsyraidsy/data (process repeatedly file by file)
2. Find all source and target. Capture with regex /(?:source|target):\s*(?:['"]([^'"]+)['"]|\[((?:['"][^'"]+['"],?\s*)+)\])/g
3. If there are no source and target, skip the file.
4. Fetch BNpcName data from xivapi, and ffxiv-datamining-ko / cn / tc
5. Map "English BNpcName" → "BNpcName ID" (works case-insensitive)
6. Map "BNpcName ID" → "localized BNpcName"

Handling edge cases

New Data	Multiple Candidates	Existing Translation	Action
✓ Found	✗ Single	-	Use new data
✓ Found	✓ Multiple	✓ Exists	Keep existing + console.warn()
✓ Found	✓ Multiple	✗ None	Output 'candidate1 / candidate2' (human review)
✗ Not found	-	-	Keep existing

Example: Multiple Candidates Found

The English name "Lindwurm" has multiple Korean translations in the datamining:

• ID 130: "린드부름" (Lindwurm variant 1)
• ID 14378: "린드블룸" (Lindwurm variant 2)

If existing translation exists:

// Existing in file:
'Lindwurm': '린드블룸',

// Script finds: ['린드부름', '린드블룸']
// Action: Keep existing '린드블룸', output console.warn()

If no existing translation:

// No existing translation in file
// Script finds: ['린드부름', '린드블룸']
// Action: Output '린드부름 / 린드블룸' (requires human review)

Example: Multiple Candidates + No Existing Translation

When the script finds multiple translation candidates for a source name, but no existing translation exists in the file:

1. Output all candidates joined by " / "

   'Lindwurm': '린드부름 / 린드블룸',

2. Output console.warn() with details

   [WARNING] example.ts: Multiple candidates for 'Lindwurm' in ko: 린드부름, 린드블룸
            No existing translation found. Manual review required.

3. Human review required

   • The generated output is syntactically valid TypeScript
   • However, the regex pattern '린드부름 / 린드블룸' won't match actual game logs
   • Developer must manually choose the correct translation for that specific encounter
   • After review, update to single value: 'Lindwurm': '린드블룸',

When the script couldn't translate all source and target

missingTranslation: true, will be automatically added.

      ...(allTranslated ? [] : [`      'missingTranslations': true,`]),

Question for Deutsch

This question is resolved. Fixed by Jaehyuk-Lee@bae9985

#,Singular,Plural,Adjective,PossessivePronoun,StartsWithVowel,Unknown0,Pronoun,Article
9876,Alchemist[p] des 2. Ordens,Alchemisten[p] des 2. Ordens,6,1,0,1,0,0
9877,Ghomoro-Golem,Ghomoro-Golems,4,1,0,1,0,0
9878,Konstrukt[p] 2,Konstrukte[p] 2,4,2,0,1,2,0
9879,Granate,Granaten,3,1,0,1,1,0
9880,Patriarch[p] des 2. Ordens Za Da,Patriarchen[p] des 2. Ordens Za Da,6,1,0,1,0,0
9881,Felsspalter[p] des 2. Ordens,Felsspalter[p] des 2. Ordens,4,1,0,1,0,0
9882,Stimme[p] von Xeven,Stimme[p] von Xeven,3,2,0,1,1,0
9883,Stimme[p] von Stanik,Stimme[p] von Stanik,3,2,0,1,1,0
9884,Stimme[p] von Isolde,Stimme[p] von Isolde,3,2,0,1,1,0

Some BNpcName have [p], [a], or [t].

Candidate 1:

    {
      'locale': 'de',
      'replaceSync': {
        'Chiseled Sculpture': 'Abbild eines Mannes',
        'Ice Pillar': 'Eissäule',
        'Beastly Sculpture': 'Abbild eines Löwen',
        'Regal Sculpture': 'Abbild eines großen Löwen',
      },
    },

Candidate 2:

    {
      'locale': 'de',
      'replaceSync': {
        'Chiseled Sculpture': 'Abbild[p] eines Mannes',
        'Ice Pillar': 'Eissäule',
        'Beastly Sculpture': 'Abbild[p] eines Löwen',
        'Regal Sculpture': 'Abbild[p] eines großen Löwen',
      },
    },

Which one is the right one? Should all [x] text be removed? The gen_oopsy_timeline_replace is using Abbild eines Löwen (Candidate 1) for now.

[Jaehyuk-Lee]

This script also found out that the original translation was not exactly right.

Example:

'Shadowkeeper': 'Roi De L\'Ombre',

FIX:

'Shadowkeeper': 'roi de l\'Ombre',

netRegex works case-insensitive

resources/regexes.ts was adding i option to regex.

cactbot/resources/regexes.ts

Lines 778 to 791 in f0e09d6

	// All regexes in cactbot are case insensitive.
	// This avoids headaches as things like `Vice and Vanity` turns into
	// `Vice And Vanity`, especially for French and German. It appears to
	// have a ~20% regex parsing overhead, but at least they work.
	let modifiers = 'i';
	if (regexpString instanceof RegExp) {
	modifiers += (regexpString.global ? 'g' : '') +
	(regexpString.multiline ? 'm' : '');
	regexpString = regexpString.source;
	}
	regexpString = regexpString.replace(/\\y\{(.*?)\}/g, (match, group) => {
	return kCactbotCategories[group as keyof typeof kCactbotCategories] || match;
	});
	return new RegExp(regexpString, modifiers);

So this change doesn't matter.

[Bing-su]

The [x] pattern exists only in German, and it is sufficient to perform the following substitution.

{
  "[t]": "(?:der|die|das)",
  "[a]": "(?:e|er|es|en)",
  "[A]": "(?:e|er|es|en)",
  "[p]": ""
}

• ref:

  cactbot/util/gen_hunt_data.ts

  Lines 106 to 130 in f0e09d6

  	const deLocaleSubstitutions = (replaceString: string): string | string[] => {
  	const substitutionMap: { [param: string]: string[] } = {
  	'[t]': ['der', 'die', 'das'],
  	'[a]': ['e', 'er', 'es'],
  	'[A]': ['e', 'er', 'es'],
  	};
  	log.debug(`Doing 'de' locale substitutions on: ${replaceString}`);
  	replaceString = replaceString.replace('[p]', '');
  	let results: string[] = [replaceString];
  	Object.keys(substitutionMap).forEach((match: string) => {
  	const newArray: string[] = [];
  	for (const name of results) {
  	if (name.includes(match))
  	substitutionMap[match]?.forEach((value: string) => {
  	newArray.push(name.replace(match, value));
  	});
  	results = newArray.length > 0 ? newArray : results;
  	}
  	});
  	if (results.length === 1)
  	return replaceString;
  	return results;
  	};

[Bing-su]

I recommend considering exceptions for cases where the results retrieved via the API start with _rsv_.

[Jaehyuk-Lee]

I didn't implement that exception point because the script can't find the "BNpcName ID" with rsv data.

1. Search and process all data files from ui/oopsyraidsy/data
2. Find all source and target. Capture with regex /(?:source|target):\s*(?:['"]([^'"]+)['"]|\[((?:['"][^'"]+['"],?\s*)+)\])/g
3. If there are no source and target, skip the file.
4. Fetch BNpcName data from xivapi, and ffxiv-datamining-ko / cn / tc
5. Map "English BNpcName" → "BNpcName ID" ← If the name is written in rsv, the script can't find matching ID.
6. Map "BNpcName ID" → "localized BNpcName"

_rsv_ data will remain untranslated, and the script will mark missingTranslation: true.

And I don't think there will be cases where only Korean (or other languages) uses _rsv_ data while English uses plain text because the ffxiv version across all region is going to be the same soon.

[Jaehyuk-Lee]

The [x] pattern exists only in German, and it is sufficient to perform the following substitution.

👍 I could find some examples from raidboss data.

        'Rebirth Of Bardam The Brave': 'Bardams Wiedergeburt',
        'Looming Shadow': 'Lauernd(?:e|er|es|en) Schatten',

I'll implement the same logic here.

Fixed by Jaehyuk-Lee@bae9985

[Jaehyuk-Lee]

I implemented the regex when the translation has multiple candidates.

Before:

'Lindwurm': '린드부름 / 린드블룸', // human review must be done

After:

'Lindwurm': '(?:린드부름|린드블룸)', // human review is optional

code change:
Jaehyuk-Lee@8bcaeed

replaceSync[source] = `(?:${Array.from(candidates).sort().join('|')})`;

[Bing-su]

I didn't implement that exception point because the script can't find the "BNpcName ID" with rsv data.

1. Search and process all data files from ui/oopsyraidsy/data
2. Find all source and target. Capture with regex /(?:source|target):\s*(?:['"]([^'"]+)['"]|\[((?:['"][^'"]+['"],?\s*)+)\])/g
3. If there are no source and target, skip the file.
4. Fetch BNpcName data from xivapi, and ffxiv-datamining-ko / cn / tc
5. Map "English BNpcName" → "BNpcName ID" ← If the name is written in rsv, the script can't find matching ID.
6. Map "BNpcName ID" → "localized BNpcName"

_rsv_ data will remain untranslated, and the script will mark missingTranslation: true.

And I don't think there will be cases where only Korean (or other languages) uses _rsv_ data while English uses plain text because the ffxiv version across all region is going to be the same soon.

RSV-decrypted data is transmitted to cactbot. Meanwhile, xivapi retains the data in encrypted RSV state.

example: BNpcName r12s lindschrat

❯ curl https://v2.xivapi.com/api/sheet/BNpcName?rows=14380
{"schema":"exdschema@2:rev:a19a52d5f60f9bf497865a327debfa33a6b3bf43","version":"98ec9341db247bc3","rows":[{"row_id":14380,"fields":{"Singular":"_rsv_14380_-1_1_0_0_S64755250_E64755250"}}]}

❯ curl https://v2.xivapi.com/api/sheet/BNpcName?rows=14380&language=de
{"schema":"exdschema@2:rev:a19a52d5f60f9bf497865a327debfa33a6b3bf43","version":"98ec9341db247bc3","rows":[{"row_id":14380,"fields":{"Singular":"_rsv_14380_-1_2_0_0_S64755250_E64755250"}}]}

❯ curl https://v2.xivapi.com/api/sheet/BNpcName?rows=14380&language=fr
{"schema":"exdschema@2:rev:a19a52d5f60f9bf497865a327debfa33a6b3bf43","version":"98ec9341db247bc3","rows":[{"row_id":14380,"fields":{"Singular":"_rsv_14380_-1_3_0_0_S64755250_E64755250"}}]}

❯ curl https://v2.xivapi.com/api/sheet/BNpcName?rows=14380&language=ja
{"schema":"exdschema@2:rev:a19a52d5f60f9bf497865a327debfa33a6b3bf43","version":"98ec9341db247bc3","rows":[{"row_id":14380,"fields":{"Singular":"_rsv_14380_-1_0_0_0_S64755250_E64755250"}}]}

❯ duckdb :memory: "SELECT Singular FROM 'https://raw.githubusercontent.com/Ra-Workspace/ffxiv-datamining-ko/refs/heads/refactor/csv/BNpcName.csv' WHERE \"#\" = 14380"
┌─────────────────────────────────────────┐
│                Singular                 │
│                 varchar                 │
├─────────────────────────────────────────┤
│ _rsv_14380_-1_6_0_0_S64755250_E64755250 │
└─────────────────────────────────────────┘

[Jaehyuk-Lee]

RSV-decrypted data is transmitted to cactbot. Meanwhile, xivapi retains the data in unencrypted RSV state.

Yes, I mean, assume there is a trigger like this.

    {
      id: 'R12S The Fixer',
      type: 'StartsUsing',
      netRegex: { id: 'B4D7', source: 'Lindschrat', capture: false },
      durationSeconds: 4.7,
      response: Responses.bigAoe('alert'),
    },

This script tries to find Lindschrat from xivapi, but it can't find one.
Because it's written as _rsv_14380_-1_1_0_0_S64755250_E64755250 instead of Lindschrat in the 'BNpcName' sheet.

Script fails to make a replaceSync, it finds some translation is missing, and adds 'missingTranslations': true,.

You can't auto-translate rsv data. So, the script let the developer to translate it manually.

[Bing-su]

You're right. I misunderstood. I'm sorry.

[Jaehyuk-Lee]

Instead, I added explicit alert log for missing translations via commit Jaehyuk-Lee@4c35443. You'll get the log like this:

Alert: ui/oopsyraidsy/data/05-shb/raid/e10s.ts: Missing translation for 'Lindschrat' in fr
Alert: ui/oopsyraidsy/data/05-shb/raid/e10s.ts: Missing translation for 'Lindschrat' in ja
Alert: ui/oopsyraidsy/data/05-shb/raid/e10s.ts: Missing translation for 'Lindschrat' in cn
Alert: ui/oopsyraidsy/data/05-shb/raid/e10s.ts: Missing translation for 'Lindschrat' in tc

I added a sample trigger with source: 'Lindschrat' in e10s.ts to test this log.
I manually added Lindschrat translation to de and ko before running the script, those are not logged as missing translation.