Automatic model downloading - Part 1 #110

motin · 2021-04-22T08:58:05Z

Fixes #5

Changes made to bergamot-translator can be reviewed here: mozilla/bergamot-translator#45

motin · 2021-04-22T08:58:29Z

Currently done:

Host models locally during development
Download models, vocabularies and lexical shortlists automatically in the extension
Make sure that translation works when using the automatically downloaded model files
Don't re-download if the model files have already been downloaded once in the current browser session

motin · 2021-04-22T12:38:33Z

Currently done:

Persist downloaded model files across browser restarts so that the user only needs to download a specific language pair once

Left to do:

Download gzipped model files so that there is less to download
Verify the hash of downloaded model files after download
Measure the time it takes to download model files
Add tests for model downloading
Show model downloading progress in translation infobar
Handle and prevent anticipated model download errors
Configure production-builds of the extension to use a remote base url for model downloads

…es and lexical shortlists if not already loaded

…asses on a clean checkout and subsequent builds will have the bergamot-models already downloaded

…imported from elsewhere

…age pair works

…trol over what and when to cache downloaded model files

…e the time it takes to download model files

…r tracking of total bytes transferred

motin · 2021-04-27T07:56:56Z

Done since last update:

Download gzipped model files so that there is less to download
Verify the hash of downloaded model files after download
Measure the time it takes to download model files
Add tests for model downloading

Screenshot showing that gzipped model files are downloaded:

Left to do:

Show model downloading progress in translation infobar
Handle and prevent anticipated model download errors
Publish available models, vocabularies and lexical shortlists somewhere extension users can access them
Configure extension to use the non-local models base url for production-build model downloads

…ation since it leads to a broken web worker bundle (webpack/webpack#7879)

motin · 2021-04-27T15:42:23Z

The task Configure production-builds of the extension to use a remote base url for model downloads is currently blocked by the fact that GitHub, where the models reside now, is not configured to serve gzipped contents transparently.

EDIT: Filed as a separate issue

…ss) + Don't attempt to persist model files if remaining quota suggest that it won't work

…mported locally

…the reverse direction

motin · 2021-04-28T11:13:09Z

Done since last update:

Handle and prevent anticipated model download errors

Left to do:

Show model downloading progress in translation infobar

I filed a separate issue for Configure extension to use the non-local models base url for production-build model downloads so that we can review and merge the general automatic model download functionality in this PR without being blocked.

…yed as text in the translation infobar

motin · 2021-04-29T09:19:39Z

Done since last update:

Show model downloading progress in translation infobar

Download.progress.infobar.prototype.UI.mov

motin · 2021-04-29T10:49:06Z

Done since last update:

Record model download time metric in post-translation telemetry

This PR is now ready for review.

motin · 2021-05-03T12:48:54Z

This PR uses a Cache API polyfill due to https://bugzilla.mozilla.org/show_bug.cgi?id=1575625. It works well enough to have it as part of this PR during review, but I have created a separate issue for eventually removing this polyfill when/if it is no longer necessary: https://github.com/mozilla-extensions/bergamot-browser-extension/issues/116. Please review the current PR as is, knowing that a follow-up PR will address the polyfill issue.

andrenatal · 2021-05-03T18:19:11Z

r+ with the condition that the polyfill is removed in the short term

motin mentioned this pull request Apr 22, 2021

Enable workerfs + don't bundle models mozilla/bergamot-translator#44

Closed

motin mentioned this pull request Apr 22, 2021

Changes for automatic model downloading in bergamot browser extension mozilla/bergamot-translator#45

Closed

motin added 5 commits April 26, 2021 09:01

Automatically download the appropriate translation models, vocabulari…

7de4ebb

…es and lexical shortlists if not already loaded

Configure bergamot models base url based on development build or not

e2be63a

Invalidate circle ci cache so that we can be sure that circle tests p…

9c74c86

…asses on a clean checkout and subsequent builds will have the bergamot-models already downloaded

Use typescript for the translation worker integration code as well

573881d

Only download model files if not already previously downloaded

f106abd

motin force-pushed the automatic-model-downloading branch from c98abde to f106abd Compare April 26, 2021 06:01

motin added 13 commits April 26, 2021 09:48

Add notice in bergamot-translator-worker.ts about the contents being …

02f1117

…imported from elsewhere

Add some type declarations to translation worker

33f2d05

Move out model download code to separate function

ef31f66

Add license headers

f874c57

Add simple in-browser test for model downloading

e8d69dd

Add test that verifies that downloaded model files caching works

cd8e702

Add test that verifies that downloading model files for another langu…

f8fb3a5

…age pair works

Replace cache.add with fetch+cache.put combo so that we have more con…

4d6d9ed

…trol over what and when to cache downloaded model files

Add fetch progress callback, track total bytes transferred and measur…

d08297e

…e the time it takes to download model files

Don't use third party lib for fetch progress callbacks + Ensure prope…

e7798cd

…r tracking of total bytes transferred

Serve model files gzipped from local http server

0cdd456

Handle missing language pair download errors gracefully

12287e9

Verify the hash of downloaded model files

0dd3d0f

motin added 2 commits April 27, 2021 11:01

Invalidate circle ci cache

b1859bb

Exclude 'translation-worker' webpack target from split chunks optimiz…

c95abb9

…ation since it leads to a broken web worker bundle (webpack/webpack#7879)

motin added 3 commits April 28, 2021 10:23

Add file sizes to model registry (useful for tracking download progre…

7f06603

…ss) + Don't attempt to persist model files if remaining quota suggest that it won't work

Add a few additional sanity/error checks

35963dd

Add in-browser test for failing download integrity checks

088526a

motin added 7 commits April 28, 2021 11:11

Cleanup model download code

77a9ab9

Catch more load model errors

e56eb82

Pin bergamot-models revision so that the expected set of models are i…

15749ad

…mported locally

Clean up import-bergamot-translator.sh

abc3c54

Restored translation for language pairs that re-use vocab files from …

a56fa8f

…the reverse direction

Cleanup and tweaks

19dac85

Summarize periodical updates on total language pair download progress

bdb7d87

motin added 2 commits April 28, 2021 14:57

Use more already available typings in translation-worker

e8d26d2

Download progress hooked up through all abstraction layers and displa…

a2e9613

…yed as text in the translation infobar

motin added 5 commits April 29, 2021 12:20

Move translateAllFramesInTab to separate file

77c9e10

Some cleanup

0c1ad9d

Record model download time metric in post-translation telemetry

701647f

General cleanup and test improvements

00e71fa

Add model download time assertions in e2e tests

bdf2757

motin marked this pull request as ready for review April 29, 2021 10:49

motin requested review from eu9ene, abhi-agg and andrenatal April 29, 2021 10:49

motin changed the title ~~Automatic model downloading~~ Automatic model downloading - Part 1 May 3, 2021

andrenatal approved these changes May 3, 2021

View reviewed changes

motin merged commit 6564226 into mozilla-extensions:main May 4, 2021

motin deleted the automatic-model-downloading branch May 4, 2021 06:34

andrenatal pushed a commit that referenced this pull request Apr 11, 2022

Fix forms not supported message (#110)

a7d2b0e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automatic model downloading - Part 1 #110

Automatic model downloading - Part 1 #110

motin commented Apr 22, 2021 •

edited

Loading

motin commented Apr 22, 2021 •

edited

Loading

motin commented Apr 22, 2021 •

edited

Loading

motin commented Apr 27, 2021 •

edited

Loading

motin commented Apr 27, 2021 •

edited

Loading

motin commented Apr 28, 2021

motin commented Apr 29, 2021

motin commented Apr 29, 2021

motin commented May 3, 2021

andrenatal commented May 3, 2021

Automatic model downloading - Part 1 #110

Automatic model downloading - Part 1 #110

Conversation

motin commented Apr 22, 2021 • edited Loading

motin commented Apr 22, 2021 • edited Loading

motin commented Apr 22, 2021 • edited Loading

motin commented Apr 27, 2021 • edited Loading

motin commented Apr 27, 2021 • edited Loading

motin commented Apr 28, 2021

motin commented Apr 29, 2021

motin commented Apr 29, 2021

motin commented May 3, 2021

andrenatal commented May 3, 2021

motin commented Apr 22, 2021 •

edited

Loading

motin commented Apr 22, 2021 •

edited

Loading

motin commented Apr 22, 2021 •

edited

Loading

motin commented Apr 27, 2021 •

edited

Loading

motin commented Apr 27, 2021 •

edited

Loading