Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP 1844 Part 1: auto-generated human-IDs and aliases #2108

Merged
merged 168 commits into from
Mar 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
168 commits
Select commit Hold shift + click to select a range
77432d9
#1844 WIP - load and save messages per file
martin-lysk Dec 8, 2023
5111111
#1844 skip message upsert if nothing has changed
martin-lysk Dec 9, 2023
f676e96
#1844 replaces original debounced save call for updated message only
martin-lysk Dec 9, 2023
5d0f5c0
#1844 first working POC - no export on save yet, cleanup needed
martin-lysk Dec 11, 2023
200e541
fixes the watch when called multiple times
martin-lysk Dec 12, 2023
27e3bab
Merge branch 'main' into 1844-sdk-persistence-of-messages-in-project-…
martin-lysk Dec 14, 2023
9f93cef
#1844 adds remove to the repository api to allow to remove files from…
martin-lysk Dec 16, 2023
fd9f538
#1844 adds rm and stat to the NodeishFilesystemSubset type
martin-lysk Dec 16, 2023
35f7fad
#1844 batches adding of files to avoid index cash rewrites
martin-lysk Dec 16, 2023
9731a83
fixes NodeishFilesystem wrapper to only proxy public properties - nee…
martin-lysk Dec 16, 2023
7b0a49c
#1844 adds reference to forked isomorphic git to allow to speedup sta…
martin-lysk Dec 16, 2023
57a3c81
#1844 adds simple per message json persistence helper
martin-lysk Dec 16, 2023
09a2d44
#1844 introduces recursive read of the message folder to allow messag…
martin-lysk Dec 16, 2023
7551290
Merge branch 'main' into 1844-sdk-persistence-of-messages-in-project-…
martin-lysk Dec 16, 2023
4f36f55
#1844 WIP first version of human-id
martin-lysk Dec 19, 2023
5015a6d
allow paraglide to use aliases (with deprecation warning)
martin-lysk Dec 19, 2023
4ae0356
#1844 removes cache again
martin-lysk Dec 19, 2023
02bc60c
cleanup paraglide alias handling
martin-lysk Dec 19, 2023
c34aa03
#1844 WIP adds messages to the inlang folder, first working roundtrip…
martin-lysk Dec 20, 2023
a65671a
#1844 WIP first version of humanReadableHash
martin-lysk Dec 21, 2023
1b69342
#1844 WIP new message id format
martin-lysk Dec 22, 2023
261e382
removes log statements
martin-lysk Dec 22, 2023
89fa791
removes log statements
martin-lysk Dec 22, 2023
f78d35f
#1844 adds error handling for broken messages
martin-lysk Dec 22, 2023
896dc62
Merge branch 'main' into 1844-sdk-persistence-of-messages-in-project-…
martin-lysk Dec 22, 2023
7e8eb15
#1844 fix badge tests
martin-lysk Dec 22, 2023
0dc8068
#1844 adds the debouncedSave back in to export to the file we importe…
martin-lysk Dec 26, 2023
8e9afd0
synchronize load and save messages
martin-lysk Jan 8, 2024
976211e
Merge branch 'main' into 1844-sdk-persistence-of-messages-in-project-…
martin-lysk Jan 8, 2024
f2bed0e
#1844 fixes second push
martin-lysk Jan 8, 2024
e671f2e
Merge branch 'main' into 1844-sdk-persistence-of-messages-in-project-…
martin-lysk Jan 13, 2024
a0d5ea6
#1844 reload messages when plugin source files change, leanup
martin-lysk Jan 13, 2024
b748564
Merge branch 'main' into 1844-sdk-persistence-of-messages-in-project-…
martin-lysk Jan 16, 2024
da8ef84
Merge branch 'main' into 1844-sdk-persistence-of-messages-in-project-…
martin-lysk Jan 17, 2024
68edada
#1844 changes alias handling to use default instead of the plugin id
martin-lysk Jan 17, 2024
42f3dd7
Merge branch 'main' into 1844-sdk-persistence-of-messages-in-project-…
martin-lysk Jan 17, 2024
4bfabbf
Merge branch 'inlang-tab' into interims-tab-merge
martin-lysk Jan 17, 2024
6ff4334
#1844 fixes recursive folder creation in ide extension
martin-lysk Jan 17, 2024
b7f5eed
#1844 introduce parallel load for messages 795 ms -> 120 ms full cal.…
martin-lysk Jan 17, 2024
064f4ea
fixes missing rm in NodishFsWithWatcher
martin-lysk Jan 18, 2024
f931ebe
#1844 parallel load of files, and recursive folder creation
martin-lysk Jan 18, 2024
679634c
fixes fs.rm in fsWithFileWatcher
martin-lysk Jan 18, 2024
619ce22
removes timer
martin-lysk Jan 18, 2024
ed345d7
Merge branch '1844-sdk-persistence-of-messages-in-project-direcory' i…
martin-lysk Jan 23, 2024
e9fd9cb
Merge branch 'main' into 1844-sdk-persistence-of-messages-in-project-…
martin-lysk Jan 23, 2024
6e6359c
#1844 introduces file locking
martin-lysk Jan 24, 2024
93102a1
#1844 adds one message for a test on git
martin-lysk Jan 25, 2024
9186799
#1844 fixed file locking
martin-lysk Jan 25, 2024
b16cc58
fix lint
Jan 25, 2024
0be208f
#1844 adds alias export for language messages files
martin-lysk Jan 25, 2024
9092f36
Merge branch '1844-sdk-persistence-of-messages-in-project-direcory' o…
martin-lysk Jan 25, 2024
07c1a25
Merge branch 'main' into 1844-sdk-persistence-of-messages-in-project-…
martin-lysk Jan 30, 2024
d3f14a2
#1844 comments out all code that writes individual files - and adds d…
martin-lysk Jan 30, 2024
1d9b845
#1844 adds missing markInitAsCompleted call
martin-lysk Jan 30, 2024
9ddabb3
#1844 moves lockfile into the root folder instead of the messages folder
martin-lysk Jan 30, 2024
f1f920c
#1844 fixes the currentMessage comparsion and the lockfile in readMes…
martin-lysk Jan 30, 2024
b7d02b7
#1844 fixes lockfile path
martin-lysk Jan 30, 2024
3d74db9
update human-id word lists based on spreadsheet
Jan 30, 2024
82a67c2
human-id in loadProject.test
Jan 30, 2024
666dddd
fix typo, add test for human-id wordlists
Jan 31, 2024
797fd93
#1844 removes per file message persistence to cleanup
martin-lysk Feb 1, 2024
f97e1c7
#1844 make sure all saveMessagesViaPlugin calls in loadProject are se…
martin-lysk Feb 1, 2024
c951036
#1844 fixes saving deadlock, fixes test of load project
martin-lysk Feb 1, 2024
2bce3c0
#1844 making the flags per project, fixes test
martin-lysk Feb 1, 2024
046b49d
Merge branch 'main' into 1844-sdk-persistence-of-messages-in-project-…
Feb 1, 2024
149e977
make more sdk tests pass
Feb 1, 2024
44c0190
#1844 fixes tests and restructure message loading
martin-lysk Feb 2, 2024
2fae716
#1844 all SDK tests green :)
martin-lysk Feb 2, 2024
b8b5002
#1844 cleanup todos
martin-lysk Feb 2, 2024
779c55c
#1844 alias on plugins/inlang-message-format
Feb 2, 2024
9d76986
#1844 use human id in templates/plugin/src/plugin.test.ts
Feb 2, 2024
0a55dda
#1844 fix cli tests: msg alias, variant ordering
Feb 2, 2024
1a78b7b
#1844 more message alias fixes
Feb 2, 2024
bf5309a
#1844 delete ide-extension..initProject.ts (missed in 23.1 merge)
Feb 2, 2024
a03e905
#1844 fix more tests with missing alias
Feb 3, 2024
c1294a8
#1844 fix lint & formatting
Feb 3, 2024
9c855da
Merge branch 'main' into 1844-sdk-persistence-of-messages-in-project-…
Feb 5, 2024
7c86e04
remove render branch spec from lix git-proxy
Feb 5, 2024
13122b5
revert render config change
Feb 5, 2024
1e37534
Merge branch 'main' into 1844-sdk-persistence-of-messages-in-project-…
Feb 6, 2024
3611915
Merge branch 'main' into 1844-sdk-persistence-of-messages-in-project-…
Feb 8, 2024
6ae5c61
WIP message query get({ where: { alias
Feb 8, 2024
7877388
Depreciate Aliases in paraglide output
LorisSigrist Feb 9, 2024
12dc8de
Preserve JSDoc on alias reexports
LorisSigrist Feb 9, 2024
b4fce39
Deduplicate alias exporting
LorisSigrist Feb 9, 2024
4ded0cb
Add tests
LorisSigrist Feb 9, 2024
7c2a77e
Make comment more compact
LorisSigrist Feb 9, 2024
4970afc
changeset and small fix for #2212
Feb 9, 2024
bc3eccb
revert accidental change to .vscode/launch.json
Feb 9, 2024
e17d165
Merge pull request #2212 from opral/depreciate-aliases
jldec Feb 9, 2024
2db2d98
new MessageQueryApi.getByDefaultAlias(), use in ide-extension
Feb 11, 2024
aa35192
Make alias searchable in Fink
NiklasBuchfink Feb 12, 2024
6b62c63
Merge branch 'main' into 1844-sdk-persistence-of-messages-in-project-…
Feb 12, 2024
7664855
fix lint - remove console.log from paraglide compileMessage.test.ts
Feb 12, 2024
acfffc0
subscribable getByDefaultAlias() with test
Feb 12, 2024
66348e3
WIP prepare to feature flag aliases
Feb 13, 2024
898fb3c
cleanup TODOs with Martin
Feb 14, 2024
32570a7
make featureFlags boolean
Feb 14, 2024
4aa8e01
WIP tests fail - fix delete handling in loadProject
Feb 14, 2024
bbad00b
correct mock.calls index in loadProject.test
Feb 14, 2024
740a4c8
project settings featureFlags: { aliases: boolean }
Feb 14, 2024
475699f
release lockfile on save, save dirtyflags in case of deletes
Feb 15, 2024
aeaee51
Merge branch 'main' into 1844-sdk-persistence-of-messages-in-project-…
Feb 15, 2024
552a422
featureFlags only true, improve translate to not wait, no console.log
Feb 15, 2024
43603a9
Merge branch 'main' into 1844-sdk-persistence-of-messages-in-project-…
Feb 19, 2024
3152105
bring back console.logs for loadProject, fix typos.
Feb 19, 2024
6491f46
WIP for Martin - run loadMessages after save
Feb 19, 2024
72f0095
Samuel PR feedback changes
Feb 20, 2024
a6a2d9a
#1844 removes test leftovers from persistence
martin-lysk Feb 20, 2024
a9e4062
#1844 reverts the changes from the previous cherrypick - doesn't belo…
martin-lysk Feb 20, 2024
5a228d4
#1844 removes rm and stats from subset fs again
martin-lysk Feb 20, 2024
95129a4
#1844 pass errors from persistence promises (save / load plugin calls…
martin-lysk Feb 20, 2024
ddd37ec
#1844 fixes lock handing and wait for one load after a save to call i…
martin-lysk Feb 20, 2024
2ce7448
#1844 removes the previous attempt
martin-lysk Feb 20, 2024
8de078b
release lock in finally, cleanup comments
Feb 20, 2024
833322e
revert change in lix/source-code/client/package.json
Feb 20, 2024
0567164
only force loadMessages after save, if load is queued
Feb 20, 2024
a81c37f
small error handling improvements for cli translate
Feb 20, 2024
b55c3f1
#1844 adds error handling for load Messages
martin-lysk Feb 21, 2024
d4e506a
remove cli exit(0), lockfile under project.inlang, mock translate api
Feb 21, 2024
b066bab
move message load and save state into project scope
Feb 21, 2024
56c3f79
s/lockFile/lockDir/
Feb 22, 2024
867f6d2
retry ci with latest chromedriver
Feb 22, 2024
1d1d8d9
update pnpm-lock
Feb 22, 2024
2c5ed1b
Merge branch 'main' into 1844-sdk-persistence-of-messages-in-project-…
Feb 22, 2024
c212255
fix machineTranslateMessage.test.ts after merge from main
Feb 22, 2024
61c264a
WIP sdk load test - not finished yet - checking CI
Feb 24, 2024
52d4ff8
WIP failing load test
Feb 26, 2024
7cc82bf
fix clean script in /sdk/load-test/repo-i18next
Feb 26, 2024
0fe35c8
Call beforeEach with async in load.test
Feb 26, 2024
81486e8
remove lint plugins from load test
Feb 26, 2024
f152b64
console.logs and vitest.config for testing
Feb 26, 2024
456941e
simplify load-test in self-contained package
Feb 26, 2024
e614f5c
#1844 optimze diff creation on message change handling, fixes lint tr…
martin-lysk Feb 27, 2024
6f081d6
load-test changes, lint fixes
Feb 27, 2024
65be089
reduce load-test messages in git to 1 per language
Feb 27, 2024
919de4a
remove load-test message files from git, simplify clean
Feb 27, 2024
d762ace
add experimental debug logging
Feb 27, 2024
36fd53e
load-test improvements
Feb 28, 2024
aec8881
load-test throttle logs to 1 per 2s instead of filtering
Feb 28, 2024
040e20b
merge main
Feb 29, 2024
a366e4a
make tests pass
Feb 29, 2024
94e502d
bring back cli translate progress bar
Feb 29, 2024
453898d
Merge branch 'main' into 1844-sdk-persistence-of-messages-in-project-…
Feb 29, 2024
8ee9a3e
revert change to machineTranslateMesssage.test
Feb 29, 2024
811c630
s/featureFlags/experimental/
Feb 29, 2024
7e0f0d4
resolve loadProject TODOs for aliases
Feb 29, 2024
0603dc4
Merge branch 'main' into 1844-sdk-persistence-of-messages-in-project-…
Mar 1, 2024
c7a47dd
enable experimental:{aliases:true} on this repo
Mar 1, 2024
3ecbc18
remove load-test version
Mar 1, 2024
1f18c8a
Merge branch 'main' into 1844-sdk-persistence-of-messages-in-project-…
Mar 1, 2024
4837297
changeset
Mar 1, 2024
133bed7
#1844 cleanup
martin-lysk Mar 4, 2024
7b397ac
review comments
Mar 5, 2024
4322620
remove old TODO
Mar 5, 2024
e6c7f8f
Merge branch 'main' into 1844-sdk-persistence-of-messages-in-project-…
Mar 11, 2024
116b211
PR cleanup
martin-lysk Mar 11, 2024
bd4834e
removes remove from the interface again
martin-lysk Mar 11, 2024
db35542
#1844 removes the unsused hasWatcher parameter
martin-lysk Mar 11, 2024
08e1e21
#1844 cleanup removes debounce
martin-lysk Mar 11, 2024
60ca5a5
#1844 removes unused variable
martin-lysk Mar 11, 2024
dabd05d
add debounce to Fink autosave
NiklasBuchfink Mar 11, 2024
f041237
merge main
Mar 11, 2024
e8cb4b9
remove Fink todo, suppress solid/reactivity warning
Mar 11, 2024
a2b1ae2
merge main, fix pnpm-lock
Mar 12, 2024
fcfff48
merge main with updated pnpm-lock
Mar 12, 2024
9edbe82
merge main to fix esbuild issue
Mar 12, 2024
c5fc396
improve load-test mock rpc server detection
Mar 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/brave-chairs-cheat.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@inlang/paraglide-js": minor
---

paraglide deprecate aliases
23 changes: 23 additions & 0 deletions .changeset/twenty-terms-remember.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
"@inlang/project-settings": minor
"@inlang/message": minor
"@inlang/paraglide-js": minor
"vs-code-extension": minor
"@inlang/cli": minor
"@inlang/rpc": minor
"@inlang/sdk": minor
"@lix-js/client": minor
"@inlang/message-lint-rule-without-source": patch
"@inlang/message-lint-rule-missing-translation": patch
"@inlang/message-lint-rule-identical-pattern": patch
"@inlang/message-lint-rule-empty-pattern": patch
"@inlang/message-lint-rule-snake-case-id": patch
"@inlang/plugin-message-format": patch
"@inlang/plugin-next-intl": patch
"@inlang/plugin-i18next": patch
"@inlang/plugin-json": patch
"@inlang/badge": patch
---

File locking for concurrent message updates through the load/store plugin api
Auto-generated human-IDs and aliases - only with experimental: { aliases: true }
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,7 @@ it("should work with multiple resources", () => {
export const createMessage = (id: string, patterns: Record<string, Pattern | string>) =>
({
id,
alias: {},
selectors: [],
variants: Object.entries(patterns).map(([languageTag, patterns]) => ({
languageTag,
Expand Down
2 changes: 2 additions & 0 deletions inlang/source-code/cli/src/commands/lint/lint.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import { mockRepo } from "@lix-js/client"
const exampleMessages: Message[] = [
{
id: "a",
alias: {},
selectors: [],
variants: [
{
Expand All @@ -29,6 +30,7 @@ const exampleMessages: Message[] = [
},
{
id: "b",
alias: {},
selectors: [],
variants: [
{
Expand Down
26 changes: 18 additions & 8 deletions inlang/source-code/cli/src/commands/machine/translate.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,8 @@ test.runIf(process.env.GOOGLE_TRANSLATE_API_KEY)(
}
}
}
}
},
{ timeout: 10000 }
)

test.runIf(process.env.GOOGLE_TRANSLATE_API_KEY)(
Expand All @@ -82,6 +83,7 @@ test.runIf(process.env.GOOGLE_TRANSLATE_API_KEY)(
const exampleMessages: Message[] = [
{
id: "a",
alias: {},
selectors: [],
variants: [
{
Expand Down Expand Up @@ -144,11 +146,19 @@ test.runIf(process.env.GOOGLE_TRANSLATE_API_KEY)(
const messages = project.query.messages.getAll()

expect(messages[0]?.variants.length).toBe(2)
expect(messages[0]?.variants[1]?.languageTag).toBe("de")
expect(
messages[0]?.variants[1]?.pattern.some(
(value) => value.type === "VariableReference" && value.name === "username"
)
).toBeTruthy()
}
expect(messages[0]?.variants.map((variant) => variant.languageTag).sort()).toStrictEqual([
"de",
"en",
])

// eslint-disable-next-line @typescript-eslint/no-non-null-assertion -- we expected the message to exist earlier
for (const variant of messages[0]!.variants) {
expect(
variant.pattern.some(
(value) => value.type === "VariableReference" && value.name === "username"
)
).toBeTruthy()
}
},
{ timeout: 10000 }
)
18 changes: 10 additions & 8 deletions inlang/source-code/cli/src/commands/machine/translate.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
import { Command } from "commander"
import { rpc } from "@inlang/rpc"
import { getInlangProject } from "../../utilities/getInlangProject.js"
import { log } from "../../utilities/log.js"
import { log, logError } from "../../utilities/log.js"
import { type InlangProject, ProjectSettings, Message } from "@inlang/sdk"
import prompts from "prompts"
import { projectOption } from "../../utilities/globalFlags.js"
Expand Down Expand Up @@ -40,7 +40,7 @@ export const translate = new Command()
const project = await getInlangProject({ projectPath: args.project })
await translateCommandAction({ project })
} catch (error) {
log.error(error)
logError(error)
}
})

Expand All @@ -53,6 +53,7 @@ export async function translateCommandAction(args: { project: InlangProject }) {
log.error(`No inlang project found`)
return
}
const experimentalAliases = args.project.settings().experimental?.aliases

const allLanguageTags = [...projectConfig.languageTags, projectConfig.sourceLanguageTag]

Expand Down Expand Up @@ -106,20 +107,24 @@ export async function translateCommandAction(args: { project: InlangProject }) {

const rpcTranslate = async (id: Message["id"]) => {
const toBeTranslatedMessage = args.project.query.messages.get({ where: { id } })!
const logId =
`"${id}"` +
(experimentalAliases ? ` (alias "${toBeTranslatedMessage.alias.default ?? ""}")` : "")

const { data: translatedMessage, error } = await rpc.machineTranslateMessage({
message: toBeTranslatedMessage,
sourceLanguageTag,
targetLanguageTags,
})
if (error) {
logs.push(() => log.error(`Couldn't translate message "${id}": ${error}`))
logs.push(() => log.error(`Couldn't translate message ${logId}: ${error}`))
return
} else if (
translatedMessage &&
translatedMessage?.variants.length > toBeTranslatedMessage.variants.length
) {
args.project.query.messages.update({ where: { id: id }, data: translatedMessage! })
logs.push(() => log.info(`Machine translated message "${id}"`))
logs.push(() => log.info(`Machine translated message ${logId}`))
}
bar.increment()
}
Expand All @@ -133,12 +138,9 @@ export async function translateCommandAction(args: { project: InlangProject }) {
log()
}

// https://github.com/opral/monorepo/issues/1846
// https://github.com/opral/monorepo/issues/1968
await new Promise((resolve) => setTimeout(resolve, 8002))
// Log the message counts
log.success("Machine translate complete.")
} catch (error) {
log.error(error)
logError(error)
}
}
3 changes: 1 addition & 2 deletions inlang/source-code/cli/src/main.ts
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,5 @@ export const cli = new Command()
version,
},
})
// https://github.com/tj/commander.js/issues/1745
process.exit(0)
// process should exit by itself once promises are resolved
})
14 changes: 14 additions & 0 deletions inlang/source-code/cli/src/utilities/log.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,17 @@ import consola from "consola"
* log.success("Success")
*/
export const log = consola

export function logError(error: any) {
log.error(causeString(error), error)
}

// Convert error.cause into a string for logging
export function causeString(error: any) {
if (typeof error === "object" && error.cause) {
if (error.cause.errors?.length) return error.cause.errors.join(", ")
if (error.cause.code) return "" + error.cause.code
return JSON.stringify(error.cause)
}
return ""
}
Loading
Loading