Refactor internationalization

[lfcnassif]

Currently there are profiles for each language and index fields change with language. This is bad and must be changed. Will break back compatibility, so will be done for 4.0 version.

[fcscoutinho]

Será lançada uma versão do iped 3.18 com a interface em português?

[wladimirleite]

@fcscoutinho, you just need to set "locale = pt-BR" in you LocalConfig.txt.

[fcscoutinho]

Troquei para pt-BR e recebi a seguinte mensagem....

"You changed 'locale' in LocalConfig.txt but configured IPEDConfig.txt or conf/AdvancedConfig.txt in root folder! Please, restore those english config files to their defaults and set configuration in 'profiles/[lang]/default' folder!"

[lfcnassif]

Please, try a fresh "install" of Iped and just change 'locale' in LocalConfig.txt and do not change any other configuration file and say if that works.

[kraftdenker]

Ive noticed that, after indexing and generating a report,/export in Englich, if one changes language set to pt-BR in LocalConfig.txt flile into the generated report, the artifacts categories disappears of the tree, besides all interface starts to operate on pt-BR. Perhaps, does one have to process and export previously using the final target language? Am Mo., 1. Juni 2020 um 10:23 Uhr schrieb Luis Nassif < notifications@github.com>:

…

Please, try a fresh "install" of Iped and just change 'locale' in LocalConfig.txt and do not change any other configuration file and say if that works. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#18 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFVAK4WCWGKYCPCJWY2VRNTRUOTVTANCNFSM4KYBPLFA> .

-- A.M

[lfcnassif]

Yes, because currently index fields change with language (category <-> categoria), you must not change locale after processing, a lot of things will stop working.

[kraftdenker]

May be is it the case of always operate indexing in english, and operate translations to the user language selection on UI layer? What about? Am Mo., 1. Juni 2020 um 14:57 Uhr schrieb Luis Nassif < notifications@github.com>:

…

Yes, because currently index fields change with language (category <-> categoria), you must not change locale after processing, a lot of things will stop working. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#18 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFVAK4UTABGDJKZZNWPYXZDRUPTXHANCNFSM4KYBPLFA> .

-- A.M

[lfcnassif]

Yes, it is planned for version 4, because will break back compatibility. There are some details about filters and complex searches in portuguese, will need to be translated to english to work, probably updating leaf nodes of lucene query tree, after parsing search expressions.

[lfcnassif]

Will start working on this breaking change for 4.0. Category values and field names (name, size, type, path...) will be stored/indexed always in english. First 2 questions (not sure if easy/possible to implement):

1. Search box must continue working if user searches for the localized category value, example for portuguese: "category:Documentos"? Or this is just desired?
2. Search box must continue working if user searches for localized field names like "nome:windows"? Or this is just desired?

[lfcnassif]

PS: I will not reimplement lucene query text parser and they changed Query object to be immutable, also removing the clone() method. Yet not sure if it is possible to manually clone the whole query tree to perform the localization on leaf nodes...

[lfcnassif]

1. Search box must continue working if user searches for the localized category value, example for portuguese: "category:Documentos"? Or this is just desired?
2. Search box must continue working if user searches for localized field names like "nome:windows"? Or this is just desired?

I managed to get this working cloning the query tree with translated leaf nodes, actually was not difficult. Will try to finish all the details and push tomorrow.

[lfcnassif]

As part of this, profiles will not depend on language anymore. @fsicoli you configured some specific brazilian categories in RefineCategories.js script task in pt-BR profiles, they will be removed, we have some options:

1. Remove all those categories because they could result in strange false positives for other countries
2. Move them to a parent category "Brazilian Software"
3. Run only if we are in pt-BR locale (I don't like very much the idea of different results just because a different UI language was chosen)

[lfcnassif]

I think about externalizing all localization files from jar files to a conf folder to easy the translation to other languages by users. Maybe keep just the default english bundles in jar files to keep the modules working independently without problems, but externalizing them would easy the translation a lot (users will see english values for keys, portuguese is not a good starting point to translate from). Any ideas?

[lfcnassif]

externalize english files and fallback to key names if not found? Not very good but should keep modules working alone...

[wladimirleite]

As part of this, profiles will not depend on language anymore. @fsicoli you configured some specific brazilian categories in RefineCategories.js script task in pt-BR profiles, they will be removed, we have some options:

1. Remove all those categories because they could result in strange false positives for other countries
2. Move them to a parent category "Brazilian Software"
3. Run only if we are in pt-BR locale (I don't like very much the idea of different results just because a different UI language was chosen)

Just adding my two cents here, these categories include Brazilian IRPF (DEC/REC files), right?
I think they are useful in some cases, so I wouldn't remove them (option 1). Maybe leaving them commented out, so they can be enabled back, is another option...
I also don't like the option 3.

[lfcnassif]

Just adding my two cents here, these categories include Brazilian IRPF (DEC/REC files), right?

Yes!

I think they are useful in some cases, so I wouldn't remove them (option 1). Maybe leaving them commented out, so they can be enabled back, is another option...

I also don't like option 1, but it is an option :)

I also don't like the option 3.

Probably will go with option 2. Actually files are not removed from original categories when added to those specific ones, so false positives won't hurt too much. Thanks @tc-wleite for your valuable cents!

[lfcnassif]

I think about externalizing all localization files from jar files to a conf folder to easy the translation to other languages by users. Maybe keep just the default english bundles in jar files to keep the modules working independently without problems, but externalizing them would easy the translation a lot (users will see english values for keys, portuguese is not a good starting point to translate from). Any ideas?

@tc-wleite any thoughts on this?

[wladimirleite]

I think about externalizing all localization files from jar files to a conf folder to easy the translation to other languages by users. Maybe keep just the default english bundles in jar files to keep the modules working independently without problems, but externalizing them would easy the translation a lot (users will see english values for keys, portuguese is not a good starting point to translate from). Any ideas?

@tc-wleite any thoughts on this?

It seems an interesting idea, which would make translations to other languages easier.
It definitely makes sense to have the English version in a external file.
Not sure if keeping a bundled version would be really necessary...

[lfcnassif]

Not sure if keeping a bundled version would be really necessary...

Also not sure.

[lfcnassif]

Thanks Wladimir, I thought something like that.

[wladimirleite]

Hi @lfcnassif!
Testing other stuff with master branch here, I noticed a possible problem related to this issue.
Using pt-BR locale, Folders and Scanned Documents categories count (in category tree panel) is always shown as 0.

Looking at the code, I *think* it may be related to the fact that SetCategoryTask class uses the localized name of these categories, while other categories use a fixed name (in English), defined in CategoriesByTypeConfig.txt.

[lfcnassif]

Thanks for warning @tc-wleite. I will fix when I return from vacation, thank you!