New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve speed and result quality for the translation memory api #612

Closed
pascalchevrel opened this Issue Feb 9, 2016 · 0 comments

Comments

Projects
None yet
1 participant
@pascalchevrel
Member

pascalchevrel commented Feb 9, 2016

Not related to the Hackathon.

I found more perf improvements for the tm/global api and it also makes the code simpler. I also found out that with the simplification we now get a bit more results (not noticeable for big locales because looking for strings almost always returns a lot of results, but for locales that only do Gaia for example, a search that would yield 3 results now would return maybe 5 results).

One thing I wonder is if we should also set a minimum quality threshold (currently the value is set to 0, that means we return all results even if the quality is very low) to maybe a value of 50. It has no impact on the speed but maybe a higher default value would be a good thing.

@pascalchevrel pascalchevrel self-assigned this Feb 9, 2016

pascalchevrel added a commit to pascalchevrel/transvision that referenced this issue Feb 9, 2016

issue #612: Improve speed and result quality for the translation memo…
…ry api

* For a global search, don't loop through all repositories, only those a locale supports (major perf gain for small locales)
* Simplify ShowResults::getTranslationMemoryResults(), send a prepared array without entities as  keys
* Ignore empty localized strings in search results

pascalchevrel added a commit to pascalchevrel/transvision that referenced this issue Feb 9, 2016

issue #612: Improve speed and result quality for the translation memo…
…ry api

* For a global search, don't loop through all repositories, only those a locale supports (major perf gain for small locales)
* Simplify ShowResults::getTranslationMemoryResults(), send a prepared array without entities as  keys
* Ignore empty localized strings in search results

pascalchevrel added a commit to pascalchevrel/transvision that referenced this issue Feb 9, 2016

issue #612: Improve speed and result quality for global translation m…
…emory api

* For a global search, don't loop through all repositories, only those a locale supports (major perf gain for small locales)
* Simplify ShowResults::getTranslationMemoryResults(), send a prepared array without entities as keys
* Ignore empty localized strings in search results (happens when the entity is in the repo but empty)

For locales that do everything, this has no perf impact, for most locales that will give a small perf boost (most locales don't do mozilla-central), for locales that have less repositories (just started and only have Aurora for example, or only translating Gaia, or not translating Firefox for iOS, the perf gain can be significant. For Guarani for example, request times are divided by 5 on my machine compared to our current code. Same for memory use.

pascalchevrel added a commit to pascalchevrel/transvision that referenced this issue Feb 9, 2016

issue #612: Improve speed and result quality for global translation m…
…emory api

* For a global search, don't loop through all repositories, only those a locale supports (major perf gain for small locales)
* Simplify ShowResults::getTranslationMemoryResults(), send a prepared array without entities as keys
* Ignore empty localized strings in search results (happens when the entity is in the repo but empty)

For locales that do everything, this has no perf impact, for most locales that will give a small perf boost (most locales don't do mozilla-central), for locales that have less repositories (just started and only have Aurora for example, or only translating Gaia, or not translating Firefox for iOS, the perf gain can be significant).
For Guarani for example, request times are divided by 5 on my machine compared to our current code. Same for memory use.

pascalchevrel added a commit to pascalchevrel/transvision that referenced this issue Feb 9, 2016

issue #612: Improve speed and result quality for global translation m…
…emory api

* For a global search, don't loop through all repositories, only those a locale supports (major perf gain for small locales)
* Simplify ShowResults::getTranslationMemoryResults(), send a prepared array without entities as keys
* Ignore empty localized strings in search results (happens when the entity is in the repo but empty)

For locales that do everything, this has no perf impact, for most locales that will give a small perf boost (most locales don't do mozilla-central), for locales that have less repositories (just started and only have Aurora for example, or only translating Gaia, or not translating Firefox for iOS, the perf gain can be significant).
For Guarani for example, request times are divided by 5 on my machine compared to our current code. Same for memory use.

pascalchevrel added a commit that referenced this issue Feb 9, 2016

Merge pull request #613 from pascalchevrel/speed
issue #612: Improve speed and result quality for global translation memory api
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment