Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MESDK-77 Requirements for Gender and Plurals [WORKING DRAFT] #2655

Conversation

LorisSigrist
Copy link
Member

@LorisSigrist LorisSigrist commented Apr 25, 2024

please don't delete this branch - parts are still being extracted thanks - @jldec

This PR updates the internal Message AST to include all the functionality we need to implement plurals, genders and other formatting functions.

It also adds the experimental and not for public use icu2-message-format plugin which is able to load (not save) messages from icu2 files.

CI will probably freak out because the Message-Type was heavily edited, but that's alright for the moment

HOW TO TEST

  • Build versioned-interfaces/message
  • Build plugin/icu2-message-format
  • Build SDK
  • Run test in development-projects/icu2

This should log the loaded messages. However, for some reason the SDK doesn't return any values or errors. The plugin does work (can also be seen in the logs).

Copy link

changeset-bot bot commented Apr 25, 2024

🦋 Changeset detected

Latest commit: d1879fd

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 40 packages
Name Type
@inlang/message Major
@inlang/rpc Patch
@inlang/sdk Patch
@inlang/message-lint-rule-camel-case-id Patch
@inlang/message-lint-rule-empty-pattern Patch
@inlang/message-lint-rule-identical-pattern Patch
@inlang/message-lint-rule-without-source Patch
@inlang/message-lint-rule-missing-translation Patch
@inlang/message-lint-rule-snake-case-id Patch
@inlang/message-lint-rule-valid-js-identifier Patch
@inlang/importer Patch
@inlang/message-lint-rule Patch
@inlang/plugin Patch
@inlang/cli Patch
vs-code-extension Patch
@inlang/server Patch
@inlang/icu2-message-format-test Patch
@inlang/badge Patch
@inlang/doc-layout-component Patch
@inlang/github-lint-action Patch
@inlang/settings-component Patch
@inlang/telemetry Patch
@inlang/cross-sell-ninja Patch
@inlang/plugin-i18next Patch
@inlang/plugin-json Patch
@inlang/plugin-m-function-matcher Patch
@inlang/plugin-next-intl Patch
@inlang/plugin-t-function-matcher Patch
@inlang/paraglide-unplugin Patch
@inlang/paraglide-js-e2e Patch
@inlang/sdk-load-test Patch
@inlang/module Patch
next-js-testapp Patch
@inlang/sdk-multi-project-test Patch
@inlang/paraglide-rollup Patch
@inlang/paraglide-vite Patch
@inlang/paraglide-webpack Patch
@inlang/paraglide-astro Patch
@inlang/paraglide-sveltekit Patch
@inlang/paraglide-sveltekit-example Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@jldec
Copy link
Contributor

jldec commented Apr 25, 2024

@LorisSigrist
Could you add a note in the OP with how to build and run this - I was hoping to look at the structure in a debugger.
(the monorepo build freaked out when I tried it 😺)

Copy link
Contributor

github-actions bot commented Apr 26, 2024

🥷 Ninja i18n – 🛎️ Translations need to be updated

❗️ New errors in setup of project /icu2/project.inlang found

ModuleImportError Couldn't import the plugin "./../../source-code/importers/icu2-message-format/dist/index.js":

Error: ENOENT: no such file or directory, open '/home/runner/work/monorepo/monorepo/merge/inlang/source-code/importers/icu2-message-format/dist/index.js'

Error cause
ENOENT: no such file or directory, open '/home/runner/work/monorepo/monorepo/merge/inlang/source-code/importers/icu2-message-format/dist/index.js'

Stack trace
Error: ENOENT: no such file or directory, open '/home/runner/work/monorepo/monorepo/merge/inlang/source-code/importers/icu2-message-format/dist/index.js'

PluginsDoNotProvideLoadOrSaveMessagesError No plugin provides a `loadMessages()` or `saveMessages()` function

In case no plugin threw an error, you likely forgot to add a plugin that handles the loading and saving of messages. Refer to the marketplace for available plugins https://inlang.com/marketplace.

displayName,
description,
settingsSchema: PluginSettings,
loadMessages: async ({ settings, nodeishFs }) => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we may consider adding an experimental function "importMessages" that uses the a Message2 type that can be mapped to the MessageLegacy. By doing so we can bypass the compatibility problems we face at the moment - just thinking out loud will discuss with you and @jldec tomorrow

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

export type Declaration = Static<typeof Declaration>
export const Declaration = Type.Union([LocalDeclaration, InputDeclaration])

export type Translation = Static<typeof Translation>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Translation vs. Locale - I think locales is the better term. It feels also more alinged with ICU glossary compare source-locale and locale.
Since this property is used everywhere (also in fixtures for testing) this naming should be chosen wisely TBD.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Loris will doublecheck the glossary and we will collect reasons for the term here

selectors: Type.Array(Expression),
variants: Type.Array(Variant),
})

export type Message = Static<typeof Message>
export const Message = Type.Object({
Copy link
Contributor

@martin-lysk martin-lysk Apr 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets discuss the pros / cons of a separate type for the internal representation to stay backward compatible with old plugins. We can prepare the discussion tomorrow.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A new type makes sense -next steps tbd with @jldec

@martin-lysk
Copy link
Contributor

@LorisSigrist Could you add a note in the OP with how to build and run this - I was hoping to look at the structure in a debugger. (the monorepo build freaked out when I tried it 😺)

@jldec i hope we can reduce the friction we create with this pr a lot by introducing a separate type internally and map the in/out type we use in load/saveMessage to the newly created type (related related), we can discuss a possible strategy in person tomorrow.

Copy link
Contributor

@martin-lysk martin-lysk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great work @LorisSigrist - just smaller findings. Good starting point to iterate further.

@jldec I marked the places where we introduce the new AST format (naming not finalized). Create message is marked as deprecated - i will investigate together with @LorisSigrist how a first internal version - that will than be called by createMessage - would look like and what functions we need alternatives for.

}
}

export function fromLegacyMessaeg(legacyMessage: LegecyFormat.Message): AST.MessageBundle {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo

selectors: [],
variants: [
inputs: [],
translations: [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change to messages

/**
* @throws If the message cannot be represented in the legacy format
*/
export function toLegacyMessage(bundle: AST.MessageBundle): LegecyFormat.Message {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jldec this is what i meant with a mapper from the new MessageBundle type to the legacy format - also see fromLegacyPattern for the other direction

displayName,
description,
settingsSchema: PluginSettings,
importMessages: async ({ settings, nodeishFs }) => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jldec this a new function we use to keep the signature of loadMessages (soon to be deprecated) untouched

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok - I wonder if there isn't a way to feature flag the api?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -0,0 +1,120 @@
import { LanguageTag as Locale } from "@inlang/language-tag"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jldec Those types describe the messageBundle in which we represent the messages internally in.

Copy link
Contributor

@jldec jldec Apr 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rename ast to message-bundle?


export const createMessage = (id: string, patterns: Record<string, Pattern | string>) =>
/**
* @deprecated Uses Legacy Message Format
Copy link
Contributor

@martin-lysk martin-lysk Apr 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jldec here we use the LegacyFormat - we deprecate this function to allow fink / sherlock etc to adopt the new (to be defined functions) but stay backward compatible

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok - I'm a bit worried about deprecation noise in our build logs - only use if really necessary

@jldec
Copy link
Contributor

jldec commented Apr 30, 2024

@martin-lysk The new interface package and importer plugin package both feel a bit premature for the scope of this PR - Are they really necessary? What capabilities are they expected to unlock for our app developers

Why not ship a small sdk-integrated importer similar to the experimental persistence feature, just for testing.

(The same is perhaps true also for the development-project in this PR - I think I'd prefer to maintain test projects for the sdk under the sdk tree)

@martin-lysk
Copy link
Contributor

@jldec yes - agree: we will keep the final PR small.
Adding the importer intended to outline in code how we the approach enables us to add new functionality to the existing SDK while keeping the old structure functional. To get to a POC of the discussed "reactive thin layer" the importer can stay as it is.
I suggest we open up a new PR when we got an agreement with the team and types got merged.

import * as MF from "messageformat"
import { displayName, description } from "../marketplace-manifest.json"
import { PluginSettings } from "./settings.js"
import { AST } from "@inlang/messages"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@martin-lysk what is this AST object? I can't find the @inlang/messages package

@jldec
Copy link
Contributor

jldec commented May 3, 2024

Thanks @martin-lysk

I asked @LorisSigrist about the motivation for the importer and he confirmed that

We needed a way to load messages with the new AST. Since we decided to leave the Plugin API as-is for compatibility we figured we would add a load-only eqivalent for the new AST.
Unless the ICU2 Importer package is moved into the SDK itself there needs to be an Interface defined somewhere

So my preference would be to follow the pattern already shipping with the experimental persistence flag, and move that importer code inside the sdk for now. That would eliminate the complexity associated with building and loading a separate plugin while we are iterating on the persistence architecture.

@jldec
Copy link
Contributor

jldec commented May 10, 2024

closing with branch intact.

@jldec jldec closed this May 10, 2024
@github-actions github-actions bot locked and limited conversation to collaborators May 10, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants