All messages removed after upgrading to v4

[marek-saji]

Describe the bug

After upgrading from 3.17.1 to 4.2.1 and using lingui JSON files. After running lingui extract --clean all of the translations are empty.

To Reproduce

My project is using natural language as IDs.

Documentation https://lingui.dev/releases/migration-4#hash-based-message-id-generation-and-context-feature promises that:

If you use natural language as an ID, you don't need to do anything special to migrate.

Expected behavior

Either or:

• have some CLI command to migrate to new hash–based IDs in JSON file

• lingui extract detecting older format of the JSON file and use translations from those instead of marking them all as obsolete

Additional context
Add any other context about the problem here.

• jsLingui version lingui --version: 4.2.1
• Babel version npm list @babel/core: 7.21.4
• Macro support:
  • I'm using SWC with @lingui/swc-plugin
  • I'm using Babel with babel-macro-plugin
  • I'm not using macro
• Your Babel config (e.g. .babelrc) or framework you use (Create React App, NextJs, Vite):

  {
    "presets": ["next/babel"],
    "plugins": ["macros", "@babel/plugin-proposal-private-methods"]
  }
  

[thekip]

If you use natural language as an ID, you don't need to do anything special to migrate.

AND Po format.

Unfortunately, there is no workaround for nonstandard json formats. You can try to write script which would convert it for you, it should be relatively simple.

[marek-saji]

In case somebody else looks for such a script:

#!/usr/bin/env node

/* eslint-disable no-restricted-syntax, no-await-in-loop */

import crypto from 'node:crypto'
import fs from 'node:fs/promises'

// from https://github.com/lingui/js-lingui/blob/59f0eb3565611d5860c883aa42a20365392ca1d8/packages/message-utils/src/generateMessageId.ts
const UNIT_SEPARATOR = '\u001F'
function generateMessageId(msg, context = '') {
  return crypto
    .createHash('sha256')
    .update(msg + UNIT_SEPARATOR + (context || ''))
    .digest('base64')
    .slice(0, 6)
}

function printHelp() {
  process.stdout.write(
    [
      'Migrate Lingui’s JSON catalogues into new hash–based keys.',
      '',
      `Usage: ${process.argv.slice(0, 2).join(' ')} JSON_FILE...`,
      '',
    ].join('\n')
  )
}

const filePaths = process.argv.slice(2)
if (!filePaths.length) {
  printHelp()
  process.exit(64)
} else if (['-h', '--help'].includes(filePaths[0])) {
  printHelp()
  process.exit(0)
}

for (const filePath of filePaths) {
  const json = await fs.readFile(filePath, { encoding: 'utf-8' })
  const msgs = JSON.parse(json)
  const newMsgs = Object.fromEntries(
    Object.entries(msgs).map(([key, value]) => {
      if (value.message) {
        // Already in new format
        return [key, value]
      }
      const message = key
      const { translation, extractedComments: comments } = value
      return [
        generateMessageId(message),
        {
          translation,
          message,
          comments,
        },
      ]
    })
  )
  await fs.writeFile(filePath, JSON.stringify(newMsgs, null, 2))
}

process.stdout.write('Done. You probably want to run this now:\n')
process.stdout.write('npx lingui extract && npx lingui compile\n')

[marek-saji]

If you use natural language as an ID, you don't need to do anything special to migrate.

AND Po format.

Unfortunately, there is no workaround for nonstandard json formats. You can try to write script which would convert it for you, it should be relatively simple.

Not sure what you mean by “nonstandard” — I’m using JSON files with lingui format. I don’t mind having to convert them myself, but “you don't need to do anything special to migrate” is misleading.

[thekip]

In case somebody else looks for such a script:

Thanks. Would you be minded to change the migration guide and add this script as a link to gist (or any other method)?

I mean "lingui" or "minimal" is not-standardized by any organization translation format. Xliff - is standardized, Po is standardized. So there are no rules how "lingui" should serialize catalog in that formats.

[marek-saji]

Gotcha. Thanks for clarification.