feat: Crowdin import automation #12698

wackerow · 2024-04-09T22:16:16Z

Description

Adds a GitHub action cron job that runs a series of actions on the first of every month:

Downloads latest build for our project from Crowdin
Unzips contents temporarily into .crowdin/
Check which directories from Crowdin (buckets) for each locale have an approval of 100%, and form bucket import list
An upgraded Crowdin Import script accepts the desired list of bucket imports, and imports them similar to how crowdin-import.ts worked in the past.
The markdown checker is automatically run on completion, and has been modified to not just console log the results, but to save them temporarily to be posted in a PR body
Git changes are staged to unique branches by language, then committed, and a PR is opened
Language-specific markdown checker results are added to the body of the PR for review.

Tasks

Confirm logic for which buckets are in-fact approved and ready to be imported
Enable pushing these new branches to GitHub with gh auth and posting one PR per language

Related Issue

Crowdin CI/CD efforts

and affiliated utils, constants and types

track and save summary; add markdown checker step to getTranslations.ts

netlify · 2024-04-09T22:16:31Z

✅ Deploy Preview for ethereumorg ready!

Name	Link
🔨 Latest commit	`e47e883`
🔍 Latest deploy log	https://app.netlify.com/sites/ethereumorg/deploys/66312a6981e1d000087f3ad8
😎 Deploy Preview	https://deploy-preview-12698--ethereumorg.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
Lighthouse	1 paths audited Performance: 30 (🔴 down 9 from production) Accessibility: 93 (no change from production) Best Practices: 92 (🔴 down 8 from production) SEO: 100 (no change from production) PWA: - View the detailed breakdown and full score reports

To edit notification comments on pull requests, go to your Netlify site configuration.

wackerow · 2024-04-09T22:58:20Z

src/scripts/crowdin/import/README.md

Still need to clean this up, and update this with the latest flow.

This README is an alternate approach to a huge comment block inside the script file, after that script was broken up.

Of note, as of the current branch commit, we still have the old scripts/crowdin-import.ts script. This PR implements a new version of this as scripts/crowdin/import/ with main.ts being the starting point.

We should decide what we want to do with the old scripts/crowdin-import.ts script, options including:

A. Delete it in favor of the new version, which limits the manual functionality a bit
B. Add manual overrides into the new version
C. Keep both. Keep the original as-is, like a "v1", to be used in any event that we need to manually import buckets in the future; Meanwhile also keep the new one as a "v2", more tailored to the automated workflow (expects a list of buckets to be passed to it, and options are simplified)

Personally think option C could be nice, at least while we're in this transition... as to not alter the existing functioning script, while allowing us to build this new one to meet the new needs.

minimalsm · 2024-04-10T14:44:07Z

src/scripts/markdownChecker.ts

  }
+
+  console.table(summary)


Do you know if the paths/line numbers are still clickable through this? It's a nice DX feature when resolving issues

Good question... They would be if run locally, but this is not designed to be run by the user, so these would more be viewed in the GitHub Action console log... where no, they wouldn't be clickable.

Will keep this in mind... might be a matter of generating links somehow for each of these 🤔

I've semi-reverted the changes here... made a log function that will log each event to the console with the previous levels (log, warn, or error), and also tracks these in a list to be used for the PR body.

This should log them the same way as before (hence removal of the final console.table(summary), and thus should still be able to be run locally to get these clickable paths.

The PR summary will output a copy/paste-able code block with yarn markdown-checker there as a convenience/reminder.

Fetches buckets with 100% approved phrases from Crowdin; in lieu of pulling from Notion board/API

wackerow · 2024-04-10T21:21:11Z

Latest commits add in a script that fetches the approval status of each directory on Crowdin and generates the "buckets list". We need to test this, and likely debug some details, but otherwise the main thing left would be enabling the actually pushing of these branches to GitHub and opening PRs for each.

The full "action" will not run until merged into dev. May be worth orchestrating this to only work for a couple languages, then bring it in and see how it does with those before removing that restriction.

update get-translations.yml workflow

Extract for loop script into function for improved readability, uses "locale", uses --body-file, patches --title syntax

refactor: clean up postLangPRs, rm no-longer-used util functions, add decompress package types

lukassim

Looks great, thanks @wackerow!

I think this is basically ready to run as is, we can test it out at the start of May for all of the April reviews that are currently in progress

pettinarip

@wackerow nice work, overall it looks good to me! I've left, mostly, refactor ideas that I'd like to see your opinions on.

=====
Been looking at the official GH/Crowdin action integration, it looks like it does most of functionality we need. Do you know what's stopping us from using it? couldn't we build our custom logic on top of it?

.github/workflows/get-translations.yml

src/scripts/crowdin/import/utils.ts

pettinarip · 2024-04-24T09:02:23Z

src/scripts/crowdin/import/utils.ts

+ * @param repoLangCode Language code used within the repo
+ * @returns void
+ */
+export const scrapeDirectory = (


This function do more than just scrape the dir. We could break out the logic with something like this and simplify the function

function scrapeDirectory(dir, cb) { // ... ls.forEach(cb) } scrapeDirectory(dir, (path) => { // do our logic })

Having done this, we could move scrapeDirectory to somewhere in @/libs/utils since it is a reusable function.

@pettinarip Curious if you see this as blocking? This logic was ported forward from the previous crowdin-import.ts file. Any chance we could handle this refactor separately? Similar with above comment about migrating things to @/lib/utils/crowdin

No, not a blocker, was more an refactor idea to make it clearer. Got it, makes sense what you are saying! we need to investigate a bit about those aliases and ways to use them in scripts.

src/scripts/crowdin/import/utils.ts

pettinarip · 2024-04-24T09:10:05Z

src/scripts/crowdin/import/utils.ts

+ * Some language codes used in the repo differ from those used by Crowdin.
+ * This is used to convert any codes that may differ when performing folder lookup.
+ */
+export const getCrowdinCode = (code: string): string =>


Shouldn't we move some of these (if not all) functions to the @/lib/utils/crowdin file?

I'd probably only keep the summary logic here since it is tightly related to this script.

@pettinarip If we do this we just need to make sure we don't import dependencies using @ syntax anywhere in the chain... In the src/scripts folder we are typically running these with ts-node which fails when it encounters that shorthand with code: 'MODULE_NOT_FOUND' (at least with our current setup).

In src/lib we use the @ import syntax, so if we want to mix them we may need to reverse that syntax in some cases.

Got it! didn't know about that problem.

src/scripts/crowdin/translations/getApprovedBuckets.ts

pettinarip · 2024-04-24T09:13:39Z

src/scripts/crowdin/translations/getTranslations.ts

Following the pattern, I'd call this main.ts since it is the main script file that is going to be executed.

src/scripts/crowdin/translations/postLangPRs.ts

src/scripts/crowdin/translations/utils.ts

Co-authored-by: Pablo Pettinari <pettinarip@gmail.com>

Co-Authored-By: Pablo Pettinari <pettinarip@gmail.com>

wackerow · 2024-04-30T02:29:28Z

Been looking at the official GH/Crowdin action integration, it looks like it does most of functionality we need. Do you know what's stopping us from using it? couldn't we build our custom logic on top of it?

Interesting! I hadn't looked into that, but I've already got a handful of ideas for how we could iterate a v2 of this new approach. If you're okay with it I'd like to note this one for later.

corwintines · 2024-04-30T04:09:36Z

src/scripts/crowdin/import/utils.ts

+      item.endsWith(".md") ||
+      item.endsWith(".svg") ||
+      item.endsWith(".xlsx")


Suggested change

item.endsWith(".md") ||

item.endsWith(".svg") ||

item.endsWith(".xlsx")

item.endsWith(".md")

I don't think we want to commit .svg and .xlsx file formats. There is a different process for importing images that @nloureiro and @lukassim have sorted out I believe.

corwintines

Just have one comment above on the files being committed. Other than that, this seems good to go for me. I agree with @pettinarip about maybe checking out the integration, but I do think this can wait for later. This is working now and we should test this out before testing out that integration IMO.

src/scripts/crowdin/import/utils.ts

pettinarip · 2024-04-30T14:12:45Z

Been looking at the official GH/Crowdin action integration, it looks like it does most of functionality we need. Do you know what's stopping us from using it? couldn't we build our custom logic on top of it?

Interesting! I hadn't looked into that, but I've already got a handful of ideas for how we could iterate a v2 of this new approach. If you're okay with it I'd like to note this one for later.

Of course! just wanted to share that with you because it looked relevant.

pettinarip

LGTM!

Co-authored-by: Corwin Smith <cssmittys@gmail.com>

wackerow added 8 commits April 5, 2024 13:47

feat: refactor crowdin-import v2

884a501

build: yarn add decompress

878fc17

feat: init getTranslations script

e23f1ba

and affiliated utils, constants and types

refactor: update markdownChecker

bc574c8

track and save summary; add markdown checker step to getTranslations.ts

feat: initialize get-translations.yml GH workflow

3e058a1

style: add emoji, remove whitespace

17ce801

chore: remove console flags from import readme

915c43c

build: update postLangPRs.sh

f67b0d2

wackerow added Status: In Progress Work is in progress internal 🏠 labels Apr 9, 2024

github-actions bot added dependencies 📦 Changes related to project dependencies needs review 👀 tooling 🔧 Changes related to tooling of the project labels Apr 9, 2024

wackerow added dependencies 📦 Changes related to project dependencies and removed needs review 👀 dependencies 📦 Changes related to project dependencies labels Apr 9, 2024

wackerow commented Apr 9, 2024

View reviewed changes

minimalsm reviewed Apr 10, 2024

View reviewed changes

wackerow added 2 commits April 10, 2024 13:53

feat: add getApprovedBuckets.ts

a5138e5

Fetches buckets with 100% approved phrases from Crowdin; in lieu of pulling from Notion board/API

chore: getTranslations using getApprovedBuckets

67f16a2

github-actions bot added the needs review 👀 label Apr 10, 2024

wackerow added 7 commits April 10, 2024 17:40

refactor: postLangPRs to TS, remove shell version

42a2c4f

update get-translations.yml workflow

refactor: clean up and patch postLangPTs script

bc51e69

Extract for loop script into function for improved readability, uses "locale", uses --body-file, patches --title syntax

fix: move gh auth to workflow yml

13841a1

feat: enable push and PR creation

b7f0f43

refactor: use translation-buckets-dirs.json

5ba3878

feat: improve pr body output

0da50a1

refactor: clean up postLangPRs, rm no-longer-used util functions, add decompress package types

chore: clean up, use dev branch

af3d45d

wackerow added needs dev approval 🧑‍💻 Approval from a developer is needed before merging and removed Status: In Progress Work is in progress needs review 👀 labels Apr 16, 2024

wackerow changed the title ~~Crowdin monthly import automation~~ feat: Crowdin monthly import automation Apr 16, 2024

wackerow added 6 commits April 17, 2024 10:58

fix: guard check for no match

c1cbc87

chore: remove redundant BucketsList declaration

f1bdfdd

chore: add locale to preview link

5564734

debug: json format

e7a4de4

revert: use current branch over "dev" default

64b1a64

chore: clean up markdown report

f53fd61

github-actions bot added the needs review 👀 label Apr 17, 2024

wackerow requested a review from lukassim April 22, 2024 18:22

lukassim approved these changes Apr 23, 2024

View reviewed changes

wackerow mentioned this pull request Apr 23, 2024

feat: Crowdin build automation #12696

Merged

wackerow changed the title ~~feat: Crowdin monthly import automation~~ feat: Crowdin import automation Apr 23, 2024

pettinarip requested changes Apr 24, 2024

View reviewed changes

wackerow and others added 2 commits April 29, 2024 17:35

chore: add schedule comment

51267c8

Co-authored-by: Pablo Pettinari <pettinarip@gmail.com>

chore: apply suggestions from code review

462592c

Co-Authored-By: Pablo Pettinari <pettinarip@gmail.com>

corwintines self-assigned this Apr 30, 2024

corwintines reviewed Apr 30, 2024

View reviewed changes

corwintines requested changes Apr 30, 2024

View reviewed changes

corwintines reviewed Apr 30, 2024

View reviewed changes

src/scripts/crowdin/import/utils.ts Outdated Show resolved Hide resolved

pettinarip approved these changes Apr 30, 2024

View reviewed changes

chore: update variable name

e47e883

Co-authored-by: Corwin Smith <cssmittys@gmail.com>

wackerow merged commit 6c066cc into dev Apr 30, 2024
5 of 6 checks passed

wackerow deleted the crowdin-cd branch April 30, 2024 17:43

This was referenced May 8, 2024

Release candidate v8.5.2 #12924

Merged

Deploy v8.5.2 #12925

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Crowdin import automation #12698

feat: Crowdin import automation #12698

wackerow commented Apr 9, 2024 •

edited

netlify bot commented Apr 9, 2024 •

edited

wackerow Apr 9, 2024

minimalsm Apr 10, 2024

wackerow Apr 10, 2024

wackerow Apr 15, 2024

wackerow commented Apr 10, 2024

lukassim left a comment

pettinarip left a comment

pettinarip Apr 24, 2024

wackerow Apr 30, 2024

pettinarip Apr 30, 2024

pettinarip Apr 24, 2024

wackerow Apr 30, 2024

pettinarip Apr 30, 2024

pettinarip Apr 24, 2024

wackerow commented Apr 30, 2024

corwintines Apr 30, 2024

corwintines left a comment

pettinarip commented Apr 30, 2024

pettinarip left a comment

feat: Crowdin import automation #12698

feat: Crowdin import automation #12698

Conversation

wackerow commented Apr 9, 2024 • edited

Description

Tasks

Related Issue

netlify bot commented Apr 9, 2024 • edited

✅ Deploy Preview for ethereumorg ready!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wackerow commented Apr 10, 2024

lukassim left a comment

Choose a reason for hiding this comment

pettinarip left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wackerow commented Apr 30, 2024

Choose a reason for hiding this comment

corwintines left a comment

Choose a reason for hiding this comment

pettinarip commented Apr 30, 2024

pettinarip left a comment

Choose a reason for hiding this comment

wackerow commented Apr 9, 2024 •

edited

netlify bot commented Apr 9, 2024 •

edited