Bug: Cannot extract keys from built-in control flow blocks #171

austinw-fineart · 2023-11-13T07:05:56Z

Is there an existing issue for this?

I have searched the existing issues

Is this a regression?

No

Current behavior

Angular v17 introduces a new built-in control flow syntax that isn't recognized by the keys extractor.

Expected behavior

This simple block should work:

<ng-container *transloco="let t">
  @if (a > b) {
    {{ t('a is greater than b') }}
  }
</ng-container>

Please provide a link to a minimal reproduction of the bug

N/A

Transloco Config

No response

Debug Logs

No response

Please provide the environment you discovered this bug in

Transloco: 5.0.7
Transloco Keys Manager: 3.8.0
Angular: 17.0.2
Node: 20.9.0
Package Manager: npm 10.2.3
OS: Windows 11 (22H2)

Additional context

No response

I would like to make a pull request for this bug

No

The text was updated successfully, but these errors were encountered:

Celtian · 2023-11-13T07:15:44Z

I agree. This is crucial and need to be done ASAP. Otherwise I will do my own extractor...

shaharkazaz · 2023-11-13T07:47:35Z

@Celtian You are more than welcome to open a PR 👍
I won't be available in the near future as my country is at war.
This upgrade is something I started on v16 as well but it's a very big change as it requires to move to ESM package

tleveque23 · 2023-11-15T21:30:42Z

@shaharkazaz This is a pretty big issue. Is there anybody other than you, that can work on that?

shaharkazaz · 2023-11-15T21:36:37Z

@tleveque23 This is an open source, anyone from the community is welcome to contribute.

Celtian · 2023-11-15T23:23:53Z

I have already made some script. Probably this is not as powerful as this library, but I hope it helps as temporary solution.

/* eslint-disable @typescript-eslint/no-explicit-any */
import { readFileSync, writeFileSync } from 'fs-extra';
import { globStream } from 'glob';
import * as path from 'path';

interface AbstractUpdateLangConfig {
  encoding: BufferEncoding;
  defaultValue: string;
}

interface AbstractKeyStoreConfig {
  keyStore: Set<string>;
}

interface AbstractCwdConfig {
  cwd: string;
}

interface AbstractLangsConfig {
  langs: string | string[];
}

interface FindOccuranceConfig extends AbstractKeyStoreConfig {
  regex: RegExp[];
  fileContent: string;
}

interface UpdateLangConfig extends AbstractUpdateLangConfig, AbstractKeyStoreConfig {
  langPath: string;
}

interface UpdateLangsConfig
  extends AbstractUpdateLangConfig,
    AbstractKeyStoreConfig,
    AbstractCwdConfig,
    AbstractLangsConfig {}

interface ParserConfig {
  formula: (key: string) => string;
  type: 'single' | 'double' | 'both';
  coveredCases: string[];
}

interface MainConfig extends AbstractUpdateLangConfig, AbstractCwdConfig, AbstractLangsConfig {
  dryRun: boolean;
  source: string | string[];
  regex: {
    html: ParserConfig[];
    typescript: ParserConfig[];
  };
}

const flattenJson = (obj: any, parentKey: string = '', separator: string = '.'): Record<string, string> => {
  let result: Record<string, string> = {};

  for (const key in obj) {
    if (obj.hasOwnProperty(key)) {
      const newKey = parentKey ? `${parentKey}${separator}${key}` : key;

      if (typeof obj[key] === 'object' && obj[key] !== null) {
        const flattenedSubObj = flattenJson(obj[key] as any, newKey, separator);
        result = { ...result, ...flattenedSubObj };
      } else {
        result[newKey] = obj[key] as string;
      }
    }
  }

  return result;
};

const findOccurance = (config: FindOccuranceConfig): void => {
  for (const rx of config.regex) {
    let matchesMarker;
    while ((matchesMarker = rx.exec(config.fileContent)) !== null) {
      const key = matchesMarker.groups?.['content']?.trim();
      if (key) {
        config.keyStore.add(key);
      }
    }
  }
};

const updateLang = (config: UpdateLangConfig): void => {
  const langFileContent = readFileSync(config.langPath).toString();

  const flattenedLang = flattenJson(JSON.parse(langFileContent));

  const langKeyStore = new Map<string, string>();

  for (const [k, v] of Object.entries(flattenedLang)) {
    langKeyStore.set(k, v);
  }

  const result: Record<string, any> = {};

  for (const key of Array.from(config.keyStore).sort((a, b) => a.localeCompare(b, 'en'))) {
    const keys = key.split('.');
    let currentObj: Record<string, any> = result;

    for (let i = 0; i < keys.length; i++) {
      const currentKey = keys[i];
      currentObj = currentObj[currentKey] =
        currentObj[currentKey] || (i === keys.length - 1 ? langKeyStore.get(key) || config.defaultValue : {});
    }
  }

  writeFileSync(config.langPath, JSON.stringify(result, null, 2), {
    encoding: config.encoding,
  });
};

const updateLangs = (config: UpdateLangsConfig): void => {
  console.log('\n✔ Writing result into language files');

  const files: string[] = [];

  const langFilesStream = globStream(config.langs, {
    cwd: config.cwd,
  });

  langFilesStream.on('data', async (langPath) => {
    files.push(langPath);

    updateLang({
      defaultValue: config.defaultValue,
      encoding: config.encoding,
      keyStore: config.keyStore,
      langPath: path.join(config.cwd, langPath),
    });
  });

  langFilesStream.on('end', () => {
    console.log(`ℹ Keys were were updated in:\n`);

    console.table(files);

    console.log('\n🌵 Done! 🌵\n');
  });
};

const main = async (config: MainConfig): Promise<void> => {
  console.log('Starting Translation Files Build 👷🏗');

  console.log('\n✔ Extracting Template and Component Keys 🗝');

  const keyStore = new Set<string>();
  let filesCount = 0;

  const rxSingleQuotes = /'(?<content>([^'\s]|\\')+)'/;
  const rxDoubleQuotes = /"(?<content>([^"\s]|\\")+)"/;

  const filesStream = globStream(config.source, {
    cwd: config.cwd,
  });

  filesStream.on('data', async (filePath) => {
    filesCount++;
    const fileContent = readFileSync(path.join(config.cwd, filePath), {
      encoding: config.encoding,
    }).toString();

    const createRegex = (parser: ParserConfig): RegExp[] => {
      if (parser.type === 'single') {
        return [new RegExp(parser.formula(rxSingleQuotes.source), 'g')];
      } else if (parser.type === 'double') {
        return [new RegExp(parser.formula(rxDoubleQuotes.source), 'g')];
      }
      return [
        new RegExp(parser.formula(rxSingleQuotes.source), 'g'),
        new RegExp(parser.formula(rxDoubleQuotes.source), 'g'),
      ];
    };

    if (filePath.endsWith('.ts')) {
      findOccurance({
        regex: [...config.regex.typescript.map(createRegex).flat()],
        fileContent,
        keyStore,
      });
    } else if (filePath.endsWith('.html')) {
      findOccurance({
        regex: [...config.regex.html.map(createRegex).flat()],
        fileContent,
        keyStore,
      });
    }
  });

  filesStream.on('end', () => {
    console.log(`ℹ ${keyStore.size} keys were found in ${filesCount} files.`);

    if (!config.dryRun) {
      updateLangs({
        cwd: config.cwd,
        defaultValue: config.defaultValue,
        encoding: config.encoding,
        langs: config.langs,
        keyStore,
      });
    } else {
      console.log(`ℹ Dry run activated. Language files will not be updated.`);

      console.log('\n🌵 Done! 🌵\n');
    }
  });
};

main({
  dryRun: false,
  encoding: 'utf-8',
  defaultValue: '███',
  cwd: path.join(process.cwd(), '..'),
  source: ['projects/portal/src/app/**/*.ts', 'projects/portal/src/app/**/*.html'],
  langs: ['projects/portal/src/assets/i18n/*.json'],
  regex: {
    html: [
      {
        formula: (key: string): string => `{{\\s*${key}\\s*\\|\\s*transloco\\s*(:\\s*.*\\s*)?}}`,
        type: 'both',
        coveredCases: [
          `{{ 'uni.close' | transloco }}`,
          `{{ "uni.close" | transloco }}`,
          `{{ 'uni.close' | transloco: variable }}`,
          `{{ "uni.close" | transloco: variable }}`,
        ],
      },
      {
        formula: (key: string): string => `"${key}\\s*\\|\\s*transloco\\s*(:\\s*.*\\s*)?"`,
        type: 'single',
        coveredCases: [`"'uni.close' | transloco"`, `"'uni.close' | transloco : variable"`],
      },
      {
        formula: (key: string): string => `'${key}\\s*\\|\\s*transloco\\s*(:\\s*.*\\s*)?'`,
        type: 'double',
        coveredCases: [`'"uni.close" | transloco'`, `'"uni.close" | transloco : variable'`],
      },
    ],
    typescript: [
      {
        formula: (key: string): string => `_\\(\\s*${key}\\s*\\)`,
        type: 'both',
        coveredCases: [`_('uni.close')`, `_("uni.close")`],
      },
      {
        formula: (key: string): string => `transloco\\.translate\\(\\s*${key}\\s*(,\\s*.*\\s*)?\\)`,
        type: 'both',
        coveredCases: [
          `transloco.translate('uni.close')`,
          `transloco.translate("uni.close")`,
          `transloco.translate('uni.close', variable)`,
          `transloco.translate("uni.close", variable)`,
        ],
      },
    ],
  },
});

Deps

typescript 5
node 18

Related issue
vendure-ecommerce/ngx-translate-extract#26

shaharkazaz · 2023-11-16T08:20:47Z

@Celtian Thanks for sharing

Celtian · 2024-01-04T21:17:16Z

I have already made some script. Probably this is not as powerful as this library, but I hope it helps as temporary solution.

/* eslint-disable @typescript-eslint/no-explicit-any */
import { readFileSync, writeFileSync } from 'fs-extra';
import { globStream } from 'glob';
import * as path from 'path';

interface AbstractUpdateLangConfig {
  encoding: BufferEncoding;
  defaultValue: string;
}

interface AbstractKeyStoreConfig {
  keyStore: Set<string>;
}

interface AbstractCwdConfig {
  cwd: string;
}

interface AbstractLangsConfig {
  langs: string | string[];
}

interface FindOccuranceConfig extends AbstractKeyStoreConfig {
  regex: RegExp[];
  fileContent: string;
}

interface UpdateLangConfig extends AbstractUpdateLangConfig, AbstractKeyStoreConfig {
  langPath: string;
}

interface UpdateLangsConfig
  extends AbstractUpdateLangConfig,
    AbstractKeyStoreConfig,
    AbstractCwdConfig,
    AbstractLangsConfig {}

interface ParserConfig {
  formula: (key: string) => string;
  type: 'single' | 'double' | 'both';
  coveredCases: string[];
}

interface MainConfig extends AbstractUpdateLangConfig, AbstractCwdConfig, AbstractLangsConfig {
  dryRun: boolean;
  source: string | string[];
  regex: {
    html: ParserConfig[];
    typescript: ParserConfig[];
  };
}

const flattenJson = (obj: any, parentKey: string = '', separator: string = '.'): Record<string, string> => {
  let result: Record<string, string> = {};

  for (const key in obj) {
    if (obj.hasOwnProperty(key)) {
      const newKey = parentKey ? `${parentKey}${separator}${key}` : key;

      if (typeof obj[key] === 'object' && obj[key] !== null) {
        const flattenedSubObj = flattenJson(obj[key] as any, newKey, separator);
        result = { ...result, ...flattenedSubObj };
      } else {
        result[newKey] = obj[key] as string;
      }
    }
  }

  return result;
};

const findOccurance = (config: FindOccuranceConfig): void => {
  for (const rx of config.regex) {
    let matchesMarker;
    while ((matchesMarker = rx.exec(config.fileContent)) !== null) {
      const key = matchesMarker.groups?.['content']?.trim();
      if (key) {
        config.keyStore.add(key);
      }
    }
  }
};

const updateLang = (config: UpdateLangConfig): void => {
  const langFileContent = readFileSync(config.langPath).toString();

  const flattenedLang = flattenJson(JSON.parse(langFileContent));

  const langKeyStore = new Map<string, string>();

  for (const [k, v] of Object.entries(flattenedLang)) {
    langKeyStore.set(k, v);
  }

  const result: Record<string, any> = {};

  for (const key of Array.from(config.keyStore).sort((a, b) => a.localeCompare(b, 'en'))) {
    const keys = key.split('.');
    let currentObj: Record<string, any> = result;

    for (let i = 0; i < keys.length; i++) {
      const currentKey = keys[i];
      currentObj = currentObj[currentKey] =
        currentObj[currentKey] || (i === keys.length - 1 ? langKeyStore.get(key) || config.defaultValue : {});
    }
  }

  writeFileSync(config.langPath, JSON.stringify(result, null, 2), {
    encoding: config.encoding,
  });
};

const updateLangs = (config: UpdateLangsConfig): void => {
  console.log('\n✔ Writing result into language files');

  const files: string[] = [];

  const langFilesStream = globStream(config.langs, {
    cwd: config.cwd,
  });

  langFilesStream.on('data', async (langPath) => {
    files.push(langPath);

    updateLang({
      defaultValue: config.defaultValue,
      encoding: config.encoding,
      keyStore: config.keyStore,
      langPath: path.join(config.cwd, langPath),
    });
  });

  langFilesStream.on('end', () => {
    console.log(`ℹ Keys were were updated in:\n`);

    console.table(files);

    console.log('\n🌵 Done! 🌵\n');
  });
};

const main = async (config: MainConfig): Promise<void> => {
  console.log('Starting Translation Files Build 👷🏗');

  console.log('\n✔ Extracting Template and Component Keys 🗝');

  const keyStore = new Set<string>();
  let filesCount = 0;

  const rxSingleQuotes = /'(?<content>([^'\s]|\\')+)'/;
  const rxDoubleQuotes = /"(?<content>([^"\s]|\\")+)"/;

  const filesStream = globStream(config.source, {
    cwd: config.cwd,
  });

  filesStream.on('data', async (filePath) => {
    filesCount++;
    const fileContent = readFileSync(path.join(config.cwd, filePath), {
      encoding: config.encoding,
    }).toString();

    const createRegex = (parser: ParserConfig): RegExp[] => {
      if (parser.type === 'single') {
        return [new RegExp(parser.formula(rxSingleQuotes.source), 'g')];
      } else if (parser.type === 'double') {
        return [new RegExp(parser.formula(rxDoubleQuotes.source), 'g')];
      }
      return [
        new RegExp(parser.formula(rxSingleQuotes.source), 'g'),
        new RegExp(parser.formula(rxDoubleQuotes.source), 'g'),
      ];
    };

    if (filePath.endsWith('.ts')) {
      findOccurance({
        regex: [...config.regex.typescript.map(createRegex).flat()],
        fileContent,
        keyStore,
      });
    } else if (filePath.endsWith('.html')) {
      findOccurance({
        regex: [...config.regex.html.map(createRegex).flat()],
        fileContent,
        keyStore,
      });
    }
  });

  filesStream.on('end', () => {
    console.log(`ℹ ${keyStore.size} keys were found in ${filesCount} files.`);

    if (!config.dryRun) {
      updateLangs({
        cwd: config.cwd,
        defaultValue: config.defaultValue,
        encoding: config.encoding,
        langs: config.langs,
        keyStore,
      });
    } else {
      console.log(`ℹ Dry run activated. Language files will not be updated.`);

      console.log('\n🌵 Done! 🌵\n');
    }
  });
};

main({
  dryRun: false,
  encoding: 'utf-8',
  defaultValue: '███',
  cwd: path.join(process.cwd(), '..'),
  source: ['projects/portal/src/app/**/*.ts', 'projects/portal/src/app/**/*.html'],
  langs: ['projects/portal/src/assets/i18n/*.json'],
  regex: {
    html: [
      {
        formula: (key: string): string => `{{\\s*${key}\\s*\\|\\s*transloco\\s*(:\\s*.*\\s*)?}}`,
        type: 'both',
        coveredCases: [
          `{{ 'uni.close' | transloco }}`,
          `{{ "uni.close" | transloco }}`,
          `{{ 'uni.close' | transloco: variable }}`,
          `{{ "uni.close" | transloco: variable }}`,
        ],
      },
      {
        formula: (key: string): string => `"${key}\\s*\\|\\s*transloco\\s*(:\\s*.*\\s*)?"`,
        type: 'single',
        coveredCases: [`"'uni.close' | transloco"`, `"'uni.close' | transloco : variable"`],
      },
      {
        formula: (key: string): string => `'${key}\\s*\\|\\s*transloco\\s*(:\\s*.*\\s*)?'`,
        type: 'double',
        coveredCases: [`'"uni.close" | transloco'`, `'"uni.close" | transloco : variable'`],
      },
    ],
    typescript: [
      {
        formula: (key: string): string => `_\\(\\s*${key}\\s*\\)`,
        type: 'both',
        coveredCases: [`_('uni.close')`, `_("uni.close")`],
      },
      {
        formula: (key: string): string => `transloco\\.translate\\(\\s*${key}\\s*(,\\s*.*\\s*)?\\)`,
        type: 'both',
        coveredCases: [
          `transloco.translate('uni.close')`,
          `transloco.translate("uni.close")`,
          `transloco.translate('uni.close', variable)`,
          `transloco.translate("uni.close", variable)`,
        ],
      },
    ],
  },
});

Deps

typescript 5
node 18

Related issue vendure-ecommerce/ngx-translate-extract#26

Script moved here:
https://www.npmjs.com/package/ngx-i18n-extract-regex-cli

kekel87 · 2024-01-05T10:35:12Z

Since Angular 16 there are also Self-Closing-Tags components which do not work with this lib: #155

shaharkazaz · 2024-01-06T07:18:15Z

@kekel87 Yes that's correct.
This library needs to be migrated to ESM to support the new Angular compiler distribution.

The community is welcome to open a PR and make this change. people like to complain but not contribute.
This library and Transloco consume a lot of time but don't even get sponsored. If I have the time I might get to it one day.
In the meantime, if someone wants this feature, they are welcome to do the work.

kekel87 · 2024-01-13T16:54:37Z

@shaharkazaz I think I've succeeded, I've opened a PR.
Let's chat inside, when you can.
All the best

mackelito · 2024-02-02T15:15:42Z

any updates?

lkuendig · 2024-04-18T10:24:14Z

thanks for your work so far! transloco-keys-manager find seems to have problems to find the right missing keys too. seems this dependency needs also an update

shaharkazaz · 2024-04-19T15:22:13Z

Control flow should be supported in the latest versions.
@austinw-fineart @lkuendig Is the original issue resolved?
If there is an issue regarding find I think let's open a new detailed issue about it.

austinw-fineart · 2024-04-22T02:20:24Z

Yep, it looks to be working. Closing as fixed.

Celtian mentioned this issue Nov 17, 2023

Support for Angular 17 new control flow syntax vendure-ecommerce/ngx-translate-extract#26

Closed

kekel87 mentioned this issue Jan 13, 2024

feat: 🎸 Add support for angular 16-17 #173

Closed

2 tasks

kekel87 mentioned this issue Mar 2, 2024

feat: 🎸 Add support for self closing tag and control flow #180

Merged

3 tasks

austinw-fineart closed this as completed Apr 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug: Cannot extract keys from built-in control flow blocks #171

Bug: Cannot extract keys from built-in control flow blocks #171

austinw-fineart commented Nov 13, 2023

Celtian commented Nov 13, 2023

shaharkazaz commented Nov 13, 2023

tleveque23 commented Nov 15, 2023

shaharkazaz commented Nov 15, 2023

Celtian commented Nov 15, 2023 •

edited

Loading

shaharkazaz commented Nov 16, 2023

Celtian commented Jan 4, 2024

kekel87 commented Jan 5, 2024

shaharkazaz commented Jan 6, 2024

kekel87 commented Jan 13, 2024

mackelito commented Feb 2, 2024

lkuendig commented Apr 18, 2024

shaharkazaz commented Apr 19, 2024

austinw-fineart commented Apr 22, 2024

Bug: Cannot extract keys from built-in control flow blocks #171

Bug: Cannot extract keys from built-in control flow blocks #171

Comments

austinw-fineart commented Nov 13, 2023

Is there an existing issue for this?

Is this a regression?

Current behavior

Expected behavior

Please provide a link to a minimal reproduction of the bug

Transloco Config

Debug Logs

Please provide the environment you discovered this bug in

Additional context

I would like to make a pull request for this bug

Celtian commented Nov 13, 2023

shaharkazaz commented Nov 13, 2023

tleveque23 commented Nov 15, 2023

shaharkazaz commented Nov 15, 2023

Celtian commented Nov 15, 2023 • edited Loading

shaharkazaz commented Nov 16, 2023

Celtian commented Jan 4, 2024

kekel87 commented Jan 5, 2024

shaharkazaz commented Jan 6, 2024

kekel87 commented Jan 13, 2024

mackelito commented Feb 2, 2024

lkuendig commented Apr 18, 2024

shaharkazaz commented Apr 19, 2024

austinw-fineart commented Apr 22, 2024

Celtian commented Nov 15, 2023 •

edited

Loading