Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(app): Use i18next-resources-to-backend instead of i18next-http-backend #6717

Merged
merged 13 commits into from
May 16, 2024

Conversation

VIKTORVAV99
Copy link
Member

@VIKTORVAV99 VIKTORVAV99 commented May 3, 2024

Issue

The current NPM package we use for a i18next backend is designed for server side translations and are causing some issues with requests failing (and falling back to the index.html) when the language don't exactly match.

For example en-US would fail and then fall back to en which would work.

The current package, i18next-http-backend, also contains a dependency on cross-fetch which are causing an additional 20kb of code to be included in our index bundle bloating the app.

We also have a problem with the JSON files not being minimised due to being in the public folder.

Closes AVO-78

Description

This PR switches out the i18next-http-backend package for i18next-resources-to-backend and moves the language files to the src directory. This together will bundle the JSON files as JS files and generate dynamic imports for them instead. Since it is now passing though the vite optimization engine it ensures the files get minimised as well.

Important

Since the api relies on the same translations as the app take some extra care to double check the modifications to the Earthfiles.

This change will also help facilitate the implementation of a service worker as it will have built in cache busting which removes the need to hash the files in the service worker instead.

Preview

On master:

dist/index.html                                 10.06 kB │ gzip:   2.23 kB
dist/assets/MapWrapper-05820bd7.css             63.75 kB │ gzip:   9.07 kB
dist/assets/index-506311f0.css                  75.01 kB │ gzip:  13.07 kB
dist/assets/panelAtoms-821fdda6.js               0.43 kB │ gzip:   0.28 kB │ map:     0.38 kB
dist/assets/units-66d4a271.js                    0.58 kB │ gzip:   0.32 kB │ map:     1.21 kB
dist/assets/web-3ddf4961.js                      0.83 kB │ gzip:   0.43 kB │ map:     1.85 kB
dist/assets/util-eb8de305.js                     1.10 kB │ gzip:   0.62 kB │ map:     4.46 kB
dist/assets/Flag-b601f090.js                     1.15 kB │ gzip:   0.61 kB │ map:     4.03 kB
dist/assets/Modal-b6ba27e5.js                    1.19 kB │ gzip:   0.70 kB │ map:     2.74 kB
dist/assets/scales-8f88c8a9.js                   1.29 kB │ gzip:   0.59 kB │ map:     3.89 kB
dist/assets/helpers-9a3f0df2.js                  1.74 kB │ gzip:   0.77 kB │ map:     7.12 kB
dist/assets/bundle.esm-1013fcd7.js               1.84 kB │ gzip:   0.78 kB │ map:    12.00 kB
dist/assets/SettingsModal-b43b7a37.js            1.94 kB │ gzip:   1.01 kB │ map:     6.76 kB
dist/assets/formatting-49b2a646.js               2.53 kB │ gzip:   0.99 kB │ map:    10.30 kB
dist/assets/FAQModal-b283678a.js                 2.56 kB │ gzip:   1.19 kB │ map:     5.52 kB
dist/assets/TooltipWrapper-a693a1f5.js           2.89 kB │ gzip:   1.28 kB │ map:     8.98 kB
dist/assets/InfoModal-d569752c.js                2.94 kB │ gzip:   1.14 kB │ map:     5.86 kB
dist/assets/LegendContainer-d637d0d0.js          3.56 kB │ gzip:   1.30 kB │ map:     9.84 kB
dist/assets/LeftPanel-f820103a.js                4.04 kB │ gzip:   1.71 kB │ map:     8.22 kB
dist/assets/utilities-78e47d1a.js                4.63 kB │ gzip:   1.78 kB │ map:    13.37 kB
dist/assets/index.esm-985fd05d.js                4.93 kB │ gzip:   1.88 kB │ map: 1,388.94 kB
dist/assets/TimeAxis-dddebbcb.js                 5.63 kB │ gzip:   2.66 kB │ map:    20.23 kB
dist/assets/getMapGrid-8b8a5fdc.js               6.19 kB │ gzip:   2.42 kB │ map:    56.57 kB
dist/assets/RankingPanel-9bdb5c0b.js             8.15 kB │ gzip:   2.82 kB │ map:    19.98 kB
dist/assets/MapControls-cd5afc49.js             11.57 kB │ gzip:   4.46 kB │ map: 1,977.05 kB
dist/assets/MapWrapper-36e39cb9.js              51.79 kB │ gzip:  18.18 kB │ map:   228.00 kB
dist/assets/sentry-86515053.js                  76.15 kB │ gzip:  25.90 kB │ map:   498.62 kB
dist/assets/config-6633fa8e.js                  98.96 kB │ gzip:  26.04 kB │ map:     0.10 kB
dist/assets/TimeControllerWrapper-f1b2a1b6.js  134.66 kB │ gzip:  40.91 kB │ map:   630.21 kB
dist/assets/ZoneDetails-ea9837ab.js            174.78 kB │ gzip:  54.87 kB │ map: 2,721.27 kB
dist/assets/flags-b73cdab7.js                  238.79 kB │ gzip:  55.25 kB │ map:   497.59 kB
dist/assets/radix-2af64b2d.js                  243.08 kB │ gzip:  78.46 kB │ map: 1,088.57 kB
dist/assets/index-2dcee676.js                  276.52 kB │ gzip:  89.80 kB │ map: 2,049.13 kB
dist/assets/recharts-c7f58839.js               346.50 kB │ gzip:  95.89 kB │ map: 1,569.75 kB
dist/assets/world-c3993af3.js                  722.40 kB │ gzip: 252.02 kB │ map:     0.10 kB
dist/assets/maplibre-gl-06a9e378.js            762.44 kB │ gzip: 208.25 kB │ map: 1,632.86 kB

On this branch:

dist/index.html                                 10.06 kB │ gzip:   2.23 kB
dist/assets/MapWrapper-05820bd7.css             63.75 kB │ gzip:   9.07 kB
dist/assets/index-506311f0.css                  75.01 kB │ gzip:  13.07 kB
dist/assets/panelAtoms-3cf20e6a.js               0.43 kB │ gzip:   0.29 kB │ map:     0.38 kB
dist/assets/units-66d4a271.js                    0.58 kB │ gzip:   0.32 kB │ map:     1.21 kB
dist/assets/web-85b3eb77.js                      0.83 kB │ gzip:   0.43 kB │ map:     1.85 kB
dist/assets/util-eb8de305.js                     1.10 kB │ gzip:   0.62 kB │ map:     4.46 kB
dist/assets/Flag-6b63fc93.js                     1.15 kB │ gzip:   0.61 kB │ map:     4.03 kB
dist/assets/Modal-722699d5.js                    1.19 kB │ gzip:   0.70 kB │ map:     2.74 kB
dist/assets/scales-a24b22b5.js                   1.29 kB │ gzip:   0.59 kB │ map:     3.89 kB
dist/assets/helpers-7e18fb6a.js                  1.74 kB │ gzip:   0.78 kB │ map:     7.12 kB
dist/assets/bundle.esm-1013fcd7.js               1.84 kB │ gzip:   0.78 kB │ map:    12.00 kB
dist/assets/SettingsModal-382f0658.js            1.94 kB │ gzip:   1.01 kB │ map:     6.76 kB
dist/assets/formatting-78fb559d.js               2.53 kB │ gzip:   0.99 kB │ map:    10.30 kB
dist/assets/FAQModal-47673c38.js                 2.56 kB │ gzip:   1.19 kB │ map:     5.52 kB
dist/assets/TooltipWrapper-7be2784c.js           2.89 kB │ gzip:   1.28 kB │ map:     8.98 kB
dist/assets/InfoModal-ead79d22.js                2.94 kB │ gzip:   1.14 kB │ map:     5.86 kB
dist/assets/LegendContainer-c6611629.js          3.56 kB │ gzip:   1.30 kB │ map:     9.84 kB
dist/assets/hi-dd8bfcee.js                       3.61 kB │ gzip:   2.25 kB │ map:     0.10 kB
dist/assets/LeftPanel-272fabd6.js                4.04 kB │ gzip:   1.71 kB │ map:     8.22 kB
dist/assets/zh-HK-e10095cd.js                    4.32 kB │ gzip:   2.40 kB │ map:     0.10 kB
dist/assets/utilities-b0f8f54e.js                4.63 kB │ gzip:   1.78 kB │ map:    13.37 kB
dist/assets/index.esm-00124485.js                4.93 kB │ gzip:   1.88 kB │ map: 1,388.94 kB
dist/assets/TimeAxis-5936d4c7.js                 5.63 kB │ gzip:   2.67 kB │ map:    20.23 kB
dist/assets/getMapGrid-d2984589.js               6.19 kB │ gzip:   2.42 kB │ map:    56.57 kB
dist/assets/ar-c8455559.js                       6.24 kB │ gzip:   2.66 kB │ map:     0.10 kB
dist/assets/nl-175626dc.js                       6.45 kB │ gzip:   2.46 kB │ map:     0.10 kB
dist/assets/da-3583ee7e.js                       6.87 kB │ gzip:   2.63 kB │ map:     0.10 kB
dist/assets/RankingPanel-2d166847.js             8.15 kB │ gzip:   2.82 kB │ map:    19.98 kB
dist/assets/MapControls-258a94b5.js             11.57 kB │ gzip:   4.46 kB │ map: 1,977.05 kB
dist/assets/zh-CN-cc08e5da.js                   21.76 kB │ gzip:  10.84 kB │ map:     0.10 kB
dist/assets/ko-a68fbea6.js                      23.39 kB │ gzip:  12.12 kB │ map:     0.10 kB
dist/assets/pl-44bcb14e.js                      27.66 kB │ gzip:   9.59 kB │ map:     0.10 kB
dist/assets/ru-70109065.js                      28.37 kB │ gzip:  11.60 kB │ map:     0.10 kB
dist/assets/ja-d529d086.js                      28.57 kB │ gzip:  13.81 kB │ map:     0.10 kB
dist/assets/hr-85d55578.js                      28.77 kB │ gzip:   9.91 kB │ map:     0.10 kB
dist/assets/zh-TW-8609abf4.js                   30.60 kB │ gzip:  16.19 kB │ map:     0.10 kB
dist/assets/cs-e7893bda.js                      30.86 kB │ gzip:  11.30 kB │ map:     0.10 kB
dist/assets/sk-d13afef1.js                      30.97 kB │ gzip:  11.38 kB │ map:     0.10 kB
dist/assets/vi-f73916f6.js                      31.64 kB │ gzip:  11.06 kB │ map:     0.10 kB
dist/assets/id-690dc7d4.js                      32.52 kB │ gzip:  10.64 kB │ map:     0.10 kB
dist/assets/fi-8e5b1b13.js                      36.20 kB │ gzip:  12.00 kB │ map:     0.10 kB
dist/assets/et-50d1c141.js                      37.52 kB │ gzip:  12.12 kB │ map:     0.10 kB
dist/assets/ro-ec2385d2.js                      37.99 kB │ gzip:  12.04 kB │ map:     0.10 kB
dist/assets/el-b13b59b4.js                      39.16 kB │ gzip:  13.96 kB │ map:     0.10 kB
dist/assets/sl-254b844f.js                      40.62 kB │ gzip:  13.46 kB │ map:     0.10 kB
dist/assets/sv-8f429bf9.js                      40.69 kB │ gzip:  13.63 kB │ map:     0.10 kB
dist/assets/no-NB-a9ea3406.js                   41.62 kB │ gzip:  14.02 kB │ map:     0.10 kB
dist/assets/it-6be951f7.js                      42.69 kB │ gzip:  13.44 kB │ map:     0.10 kB
dist/assets/es-bc42cb55.js                      43.99 kB │ gzip:  13.99 kB │ map:     0.10 kB
dist/assets/pt-BR-8af7dcd8.js                   44.15 kB │ gzip:  13.93 kB │ map:     0.10 kB
dist/assets/de-9ba00104.js                      47.10 kB │ gzip:  15.33 kB │ map:     0.10 kB
dist/assets/en-0854275d.js                      48.82 kB │ gzip:  15.28 kB │ map:     0.10 kB
dist/assets/fr-26f0d94a.js                      50.54 kB │ gzip:  16.14 kB │ map:     0.10 kB
dist/assets/MapWrapper-e46622fa.js              51.79 kB │ gzip:  18.19 kB │ map:   228.00 kB
dist/assets/sentry-75a44360.js                  76.15 kB │ gzip:  25.90 kB │ map:   498.62 kB
dist/assets/config-6633fa8e.js                  98.96 kB │ gzip:  26.04 kB │ map:     0.10 kB
dist/assets/TimeControllerWrapper-6607604c.js  134.66 kB │ gzip:  40.91 kB │ map:   630.21 kB
dist/assets/ZoneDetails-68e391ab.js            174.78 kB │ gzip:  54.87 kB │ map: 2,721.27 kB
dist/assets/flags-b73cdab7.js                  238.79 kB │ gzip:  55.25 kB │ map:   497.59 kB
dist/assets/radix-2af64b2d.js                  243.08 kB │ gzip:  78.46 kB │ map: 1,088.57 kB
dist/assets/index-03f54fb3.js                  258.37 kB │ gzip:  84.76 kB │ map: 1,986.48 kB
dist/assets/recharts-57d33d3e.js               346.50 kB │ gzip:  95.89 kB │ map: 1,569.75 kB
dist/assets/world-c3993af3.js                  722.40 kB │ gzip: 252.02 kB │ map:     0.10 kB
dist/assets/maplibre-gl-06a9e378.js            762.44 kB │ gzip: 208.25 kB │ map: 1,632.86 kB

Double check

  • I have run pnpx prettier@2 --write . and poetry run format in the top level directory to format my changes.

@github-actions github-actions bot added frontend 🎨 dependencies Pull requests that update a dependency file labels May 3, 2024
@VIKTORVAV99
Copy link
Member Author

Why is the python stuff using the app files... 👀

@github-actions github-actions bot added the python Pull requests that update Python code label May 3, 2024
@VIKTORVAV99 VIKTORVAV99 added performance 🏎 techdebt Unpleasantness that does (or may in future) affect development labels May 3, 2024
@VIKTORVAV99 VIKTORVAV99 marked this pull request as ready for review May 3, 2024 20:06
@VIKTORVAV99 VIKTORVAV99 changed the title chore(app): Use i18next-resources-to-backend instead of i18next-http-backend feat(app): Use i18next-resources-to-backend instead of i18next-http-backend May 3, 2024
@madsnedergaard
Copy link
Member

A lot of internal tools are using the country names from en.json. I am wondering if we should first try to separate that out, so it's available in the config somewhere instead?

Right now, any change to en.json is causing a lot of projects to rebuild which is kinda silly (unless the EN zone names change) :)

@VIKTORVAV99
Copy link
Member Author

A lot of internal tools are using the country names from en.json. I am wondering if we should first try to separate that out, so it's available in the config somewhere instead?

Right now, any change to en.json is causing a lot of projects to rebuild which is kinda silly (unless the EN zone names change) :)

Just the zone names?

I had planned on doing that as a part of a later change which would use namespaces (that means we can separate out the core translations, zone names and say the FAQ into their own files.

But would it be enough to just leave a copy of the en.json file in the old location for now?
Don't the internal projects rebuild on every change to the en.json file as it is right now anyway?

@madsnedergaard
Copy link
Member

A lot of internal tools are using the country names from en.json. I am wondering if we should first try to separate that out, so it's available in the config somewhere instead?
Right now, any change to en.json is causing a lot of projects to rebuild which is kinda silly (unless the EN zone names change) :)

Just the zone names?

I had planned on doing that as a part of a later change which would use namespaces (that means we can separate out the core translations, zone names and say the FAQ into their own files.

But would it be enough to just leave a copy of the en.json file in the old location for now? Don't the internal projects rebuild on every change to the en.json file as it is right now anyway?

When we change a UI translation in en.json (like a button label), it causes all internal projects relying on the translation file to get the full zone names (e.g. DK-DK2 -> East Denmark) to rebuild. This is wasteful as those internal projects only need the zone names, and we almost never change those anyway.

As first I thought about having them in the Python configuration files instead, but then we would have to duplicate it for the app to use. But maybe we can come up with another solution? I honestly think we could even consider having the English name for each zone directly in the zone YAML files :)

@VIKTORVAV99
Copy link
Member Author

A lot of internal tools are using the country names from en.json. I am wondering if we should first try to separate that out, so it's available in the config somewhere instead?
Right now, any change to en.json is causing a lot of projects to rebuild which is kinda silly (unless the EN zone names change) :)

Just the zone names?
I had planned on doing that as a part of a later change which would use namespaces (that means we can separate out the core translations, zone names and say the FAQ into their own files.
But would it be enough to just leave a copy of the en.json file in the old location for now? Don't the internal projects rebuild on every change to the en.json file as it is right now anyway?

When we change a UI translation in en.json (like a button label), it causes all internal projects relying on the translation file to get the full zone names (e.g. DK-DK2 -> East Denmark) to rebuild. This is wasteful as those internal projects only need the zone names, and we almost never change those anyway.

As first I thought about having them in the Python configuration files instead, but then we would have to duplicate it for the app to use. But maybe we can come up with another solution? I honestly think we could even consider having the English name for each zone directly in the zone YAML files :)

I think the namespace solution would be the best solution without duplicating anything and without breaking any translations. Then the zones names would be in their own file which would basically never change.

But I think that should be a follow up change to this PR to try and stick to the idea of perfect PRs (only one change per PR). But I'm also not totally against the idea of doing it here.

@madsnedergaard
Copy link
Member

But I think that should be a follow up change to this PR to try and stick to the idea of perfect PRs (only one change per PR). But I'm also not totally against the idea of doing it here.

I agree it should be a separate PR :)
I think we should make a PR ready for the backend to update the file location, so we are not blocking app releases for too long.

Earthfile Outdated
Comment on lines 14 to 15
COPY web/src/locales/en.json ./web/public/locales/en.json
COPY __init__.py ./__init__.py
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to update the dest path as well, and update it everywhere it's used in monorepo - otherwise there will be a discrepancy between running things on your local machine vs on CI :)

Suggested change
COPY web/src/locales/en.json ./web/public/locales/en.json
COPY __init__.py ./__init__.py
COPY web/src/locales/en.json ./web/src/locales/en.json
COPY __init__.py ./__init__.py

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do de even need to copy them then? Don't we copy the entire src folder?

My thinking here was that we could merge this without impacting other deployments and then point them to the new file location if needed. But as you say it wouldn't match local development.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other project use this +src-files target, which includes translations but not the whole app src folder. Otherwise they would rebuild all the time 😅

used like this: COPY ../../contrib+src-files/* /contrib

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh god 😅

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which is why I thought it might be better to first split it out, and then afterwards do this change 🙈

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, but splitting it out would require other changes as well...

Could we just not leave a copy in the old location for now (maybe with just the zone names) and then migrate it?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we could, then we should remove anything but zone names to avoid confusing contributors :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And also make sure we update documentation in various places

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll update the Wiki after this has been approved (but right before I merge it) so we don't cause a mismatch for any longer period of time.

Copy link

sentry-io bot commented May 9, 2024

Sentry Issue: APP-WEB-31W

scripts/utils.py Outdated Show resolved Hide resolved
@madsnedergaard
Copy link
Member

Let's wait a bit with merging this so it gets it's own release in case something in the backend acts up :)

@VIKTORVAV99
Copy link
Member Author

VIKTORVAV99 commented May 15, 2024

Note to self:
Update Wiki!

EDIT:
Wiki Updated!

@VIKTORVAV99 VIKTORVAV99 enabled auto-merge (squash) May 16, 2024 10:08
@VIKTORVAV99 VIKTORVAV99 merged commit ebd3915 into master May 16, 2024
21 checks passed
@VIKTORVAV99 VIKTORVAV99 deleted the vik/use_i18next-resources-to-backend branch May 16, 2024 10:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies Pull requests that update a dependency file frontend 🎨 infrastructure javascript Pull requests that update Javascript code performance 🏎 python Pull requests that update Python code techdebt Unpleasantness that does (or may in future) affect development translations 🗣
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants