Custom braille tables

[nvaccessAuto]

Reported by falinn.onda on 2013-06-25 11:49
Following the discussion in
http://community.nvda-project.org/ticket/3284#comment:5

I think it is a good idea that NVDA will enable users to customize tables,
if they know the table format (by referring to louise docs).

In response to jteh
" * Coming up with an elegant user interface for this is tricky.

• We then have to document and explain how it works.
• It encourages users to copy files into the NVDA program directory. In
  general, users should never touch the NVDA program directory, only their
  config directory."

jteh, i agree with both comments.
I did not mean that NVDA will add a graphical user interface or that NVDA will have full documentation about customizing tables.
I do not think that is necessary since there is already a very good table definition format defined and explained in detail by louise documentation.
I think other users can tweak tables to their needs like i was able to after reading louise documentation.

the only thing i am suggesting is to have several table files which the user will be able to edit. I agree that it is bad practice to edit the default standard tables in louise/tables and that is exactly why i think it is a good idea to have few table files in userConfig\brailletables (or similar).
The only thing i need from NVDA is to have an option in the braille setting input/output tables combo boxes which lets me choose to use those custom files i wrote.
It is the user responsibility to edit the file outside NVDA in a text editor of his choice (or get a file someone else wrote).).

For example if we can put other1_custom_user_table.ctb other2_custom_user_table.ctb and other3_custom_user_table.ctb
in userConfig\brailleTables. (i'm suggesting file names starting with o because there is no such tables that start with o right now which will make it easy to switch to those combo )entries by pressing o ).

I am aware that not everyone will understand how to edit tables but most advanced users will benefit from this option and maybe give the tables to other users in their community or publish online like addons.

In NVDA documentation we can only add a chapter titled "Customizing braille tables":
Advanced users who wish to use custom braille tables can edit the tables files x, y, z in the folder w. For a definition of the file format see luise documentation link.
Note that users who only wish to add symbols to ecisting tables can use the include command.

Here we can have a small example file that includes english braille table and adds additional letter and punctuation as example.

What do you think?
Blocking #4996

[nvaccessAuto]

Comment 1 by tyrylu on 2014-03-19 17:15
Hello,
i know that this ticket is quite old, but any progress on this front? I'd also like this feature. Some usage in my case would be a little more modified czech braille table, i think. The process of getting the nonexistent table to liblouis and then requesting it to be added to the TABLES constant in braille.py is probably too complex to do. On the other hand, i do not like overwriting some table (most commonly some which will never personally be used and also supports being a input table).

[nvaccessAuto]

Comment 4 by jteh on 2015-03-25 05:03
I'm very reluctant to do this. As I've already noted, it's difficult to do this elegantly and it'll be confusing for most users who don't need it.

I think this is add-on territory. However, as @dkager noted in ticket:4996#comment:1, we'll probably need to tweak the core slightly to make this possible from an add-on. I think two tweaks would be needed:

1. Make braille.TABLES a list instead of a tuple.
2. Allow absolute path names in braille.TABLES.

[nvaccessAuto]

Comment 5 by dkager on 2015-03-25 11:15
I'd be interested in looking at this. That is, applying the two points mentioned in comment:4 and doing some testing. I'm sure a core developer could do this much more quickly, but it would be a good excuse to get into Python for me.

I'm assuming this specific solution doesn't require its own ticket. Please correct if wrong.
(Edited to remove my noob assumption that working on the same bit of code in multiple branches would break. Git is clearly quite magical.)

[nvaccessAuto]

Comment 6 by dkager on 2015-03-26 15:24
Alright, here's a possible solution to the problem.

Replying to jteh:

1. Make braille.TABLES a list instead of a tuple.

This is easy enough, but while looking at the code I came up with some additional changes to make appending custom braille tables more convenient:

• Rename braille.TABLES to braille.tables (uppercase to lowercase), since the list will no longer be constant.
• Drop braille.INPUT_TABLES and build it on-the-fly in BrailleSettingsDialog.makeSettings. This is the only place where the list of input tables is referenced. Building it only when needed means that if you add a custom table you only have to update braille.TABLES without worrying about updating braille.INPUT_TABLES. Add-ons that need a list of input tables can simply build their own list similar to how the GUI would do it.
• While making things more dynamic, it might also be useful to sort the list of braille tables alphabetically in the GUI, as opposed to ordered by liblouis table filename. This again should be done dynamically because for maintainers the ordering by filename is actually quite useful.

Replying to jteh:

1. Allow absolute path names in braille.TABLES.

This one is a bit more interesting. My suggestion for finding custom tables is:

1. Check if the table can be read from braille.TABLES_DIR. If it does, use it. If not, go to (2).
2. Check if the table filename is a readable absolute path. If it does, use it. If not, go to (3).
3. Use the default table en-us-comp8.ctb. This ensures that (deaf-blind) users don't get stuck without braille if they switch to a table that isn't available, i.e. Currently used Danish braille tables no longer exist #4986.
4. In all cases, add braille-patterns.cti to the list of tables.
5. Use the resulting list (of exactly two items) in calls to louis.translate.
   An alternative solution would be to use a function or regex to determine whether a table path is absolute or not. In this case I would still include (3) to make sure a default table is loaded. Getting stuck with no braille (or speech) should be avoided when possible.

Currently the list of translation tables to use with louis.translate is built in braille.Region.update. At first glance this seems highly inefficient. It would only get worse if we started statting the filesystem when building the list. My initial suggestion would be to add a class attribute braille.Region._translationTables that is initialized once and only updated if the user changes the translation table in the GUI. For symmetry, brailleInput should be updated in a similar fashion.
The question is if this is acceptable. Currently add-ons can change the translation table simply by assigning to config.conf["braille"]["translationTable"]. This functionality would be lost, but with the advantage that the same list isn't rebuilt everytime the braille display is refreshed.

Thoughts? I'd like to implement this but I want to do it right.

[nvaccessAuto]

Comment 7 by leonarddr (in reply to comment 6) on 2015-03-26 21:41
Replying to dkager:

1. Use the default table en-us-comp8.ctb. This ensures that (deaf-blind) users don't get stuck without braille if they switch to a table that isn't available, i.e. Currently used Danish braille tables no longer exist #4986.

I tend to disagree. When pressing the OK button and a table couldn't be found, an error message should be displayed similar to the braille display error message. Loading a default table puts you on the wrong track.

[nvaccessAuto]

Comment 8 by dkager (in reply to comment 7) on 2015-03-27 07:53
Replying to leonarddr:

Replying to dkager:

1. Use the default table en-us-comp8.ctb. This ensures that (deaf-blind) users don't get stuck without braille if they switch to a table that isn't available, i.e. Currently used Danish braille tables no longer exist #4986.

I tend to disagree. When pressing the OK button and a table couldn't be found, an error message should be displayed similar to the braille display error message. Loading a default table puts you on the wrong track.

Thanks. That would be even better of course. And it should probably be done in more places (such as typing "hello" in the capital pitch change edit field silently converting the string to 0).
Barring further suggestions I'll go and implement these canges to see how it works out in the real world.

[nvaccessAuto]

Comment 9 by nvdakor on 2015-03-27 15:41
Hi,
Ah, checking values before submitting to the config file? I think that's a separate ticket (there is another ticket which suggests that nV Access use ConfigObj 5.x.y).
as for table verification, I agree with Leonard. For now, I think we should concentrate on braille tables, and expand it to other edit fields in a separate ticket (that ticket may take care of modernizing config format).
Thanks.

[nvaccessAuto]

Comment 12 by dkager on 2015-03-29 12:37
Continuing the brainstorming...
I implemented the suggested changes in this branch. See this file for an example using this new code based on #3470. The major shortcoming of this solution is that there is still no mechanism to verify if the selected table exists before switching to it.

This patch works, but I would really like to factor all of the table code out into a new module brailleTableHandler. Initial responsibilities of this module:

• Maintain the list of available tables (currently braille.TABLES).
• Provide the list of input tables as a function (as opposed to the current approach of computing it once and storing it in braille.INPUT_TABLES).
• Provide the location of the stock tables (currently braille.TABLES_DIR).
• Provide functions to add the default tables directory to relative paths and to verify if a table exists in the filesystem.
• Cache the default tables list so it doesn't have to be recreated on every braille.Region.update and brailleInput.BrailleInputHandler.input.

Some interesting future use cases for brailleTableHandler:

• Provide language-to-table mapping for at least the major languages and tables. As @bramd pointed out this can be very useful for doing automatic language switching (e.g. display Russian webpages with a Russian table).
• Cache multiple lists of tables so you can decide which table to use on a per-Region basis without too much overhead.

Another change I would like to bring up for discussion is to provide a standardized place for add-ons to store custom tables. The rationale behind this is that distinguishing tables based on absolute versus relative paths isn't intuitive. For example, it seems convenient to write the following lines to append a custom table:

table = os.path.join(globalVars.appArgs.configPath, "custom.ctb")
braille.tables.append((table, _("My custom table"), True))

This won't work because configPath doesn't have to be absolute. You can add os.path.abspath, but that isn't directly apparent. Nor can you write paths like ..\..\my_tables\custom.ctb because the current code would go search in braille.TABLES_DIR.
What if we add a folder louis\tables to the user config directory? Then brailleTableHandler could go and look for tables there, either based on a new boolean column in the table definitions or simply by first trying the user config and falling back to stock if the table isn't found there.

Of course there are other ways to address the issue. Having implemented a basic fix I can't say I'm satisfied with the results. Maybe @jteh could shed some light on what is possible/allowed/desirable. One major consequence of moving the table definitions into their own file is that existing patches will likely break. On the other hand, such a change would improve maintainability even if support for custom tables isn't added.

[nvaccessAuto]

Comment 13 by jteh (in reply to comment 12) on 2015-03-30 02:28
I haven't actually looked at your code yet. However, some replies:

Replying to dkager:

I would really like to factor all of the table code out into a new module brailleTableHandler.

I agree we need more advanced table handling, but I think it can just go in the braille module, unless you have strong reasons to split it into a separate module.

Initial responsibilities of this module:

• Provide the list of input tables as a function (as opposed to the current approach of computing it once and storing it in braille.INPUT_TABLES).

Makes sense.

• Provide functions to add the default tables directory to relative paths

Makes sense, though this will probably only be called when caching.

and to verify if a table exists in the filesystem.

Ideally, this wouldn't be necessary for in-built tables. We should probably make the build system verify that all referenced in-built tables exist and fail the build if they don't. However, it's definitely necessary for add-on tables.

• Cache the default tables list so it doesn't have to be recreated on every braille.Region.update and brailleInput.BrailleInputHandler.input.

Makes sense.

Some interesting future use cases for brailleTableHandler:

• Provide language-to-table mapping for at least the major languages and tables.

A while ago, I was thinking that language information should be stored in the tables data structure, which would require refactoring it somewhat. This way, you don't have to define information for a table in multiple places. However, because a table can actually support multiple regions, perhaps it makes more sense to have a separate language map as you suggest. Hmm.

Another change I would like to bring up for discussion is to provide a standardized place for add-ons to store custom tables. ...

I do agree there needs to be an easy way for add-ons to do this, but:

What if we add a folder louis\tables to the user config directory?

I'm not keen on this idea because it means add-ons will be placing data (other than configuration) outside their add-on directory. That makes them much harder to remove and means data could be left behind.

Note that it's already possible for an add-on to get its directory from its own code: addonHandler.getCodeAddon().path. However, I guess we could also consider a utility function for an add-on to add a table.

This won't work because configPath doesn't have to be absolute.

Which is something that has bitten us in the past and something we probably should fix, but that's a separate issue.

[nvaccessAuto]

Comment 14 by jteh (in reply to comment 6) on 2015-03-30 02:35
Replying to dkager:

• While making things more dynamic, it might also be useful to sort the list of braille tables alphabetically in the GUI, as opposed to ordered by liblouis table filename.

Makes sense.

Currently the list of translation tables to use with louis.translate is built in braille.Region.update. At first glance this seems highly inefficient.

It's not really an issue now, but as you say, any increased complexity requires caching.

My initial suggestion would be to add a class attribute braille.Region._translationTables that is initialized once and only updated if the user changes the translation table in the GUI.

I'd probably prefer it was an instance variable on braille.BrailleHandler, primarily because that's also where the active display is set.

The question is if this is acceptable. Currently add-ons can change the translation table simply by assigning to config.conf["braille"]["translationTable"]. This functionality would be lost, but with the advantage that the same list isn't rebuilt everytime the braille display is refreshed.

Backwards incompatible changes are never ideal, but I think this one is necessary for performance if we implement custom tables, etc. You should make sure to refresh the cache in braille.BrailleHandler.handleConfigProfileSwitch.

[nvaccessAuto]

Comment 15 by jteh (in reply to comment 7) on 2015-03-30 02:37
Replying to leonarddr:

1. Use the default table en-us-comp8.ctb. This ensures that (deaf-blind) users don't get stuck without braille if they switch to a table that isn't available, i.e. Currently used Danish braille tables no longer exist #4986.

I tend to disagree. When pressing the OK button and a table couldn't be found, an error message should be displayed similar to the braille display error message.

That's fine, but there still needs to be a fall back to a default table if, for example, NVDA is started with a missing table. We don't want a user to completely lose braille in that case.

[nvaccessAuto]

Comment 16 by dkager (in reply to comment 13) on 2015-03-30 19:08
Replying to jteh:

Replying to dkager:

I would really like to factor all of the table code out into a new module brailleTableHandler.

I agree we need more advanced table handling, but I think it can just go in the braille module, unless you have strong reasons to split it into a separate module.

Based on your later suggestion in comment:14 to put table handling in braille.BrailleHandler:

1. Region doesn't currently know about BrailleHandler (at least not explicitly). Should this be changed? And if so, how?
2. Similarly, BrailleInputHandler would have to know BrailleHandler.
3. A separate handler class seems more independent. Handling tables in BrailleHandler suggests that tables are more about translation/output than about input.
   Of course one could add a new class to the braille module, thereby not adding another file/module. Actually, my initial implementation used BrailleHandler, but I got stuck coming up with a good solution to (1) and (2).

and to verify if a table exists in the filesystem.

Ideally, this wouldn't be necessary for in-built tables. We should probably make the build system verify that all referenced in-built tables exist and fail the build if they don't. However, it's definitely necessary for add-on tables.

I'd say better safe than sorry, i.e. have a fallback table for all situations. Should make the logic slightly less complex too.

• Provide language-to-table mapping for at least the major languages and tables.

A while ago, I was thinking that language information should be stored in the tables data structure, which would require refactoring it somewhat. This way, you don't have to define information for a table in multiple places. However, because a table can actually support multiple regions, perhaps it makes more sense to have a separate language map as you suggest. Hmm.

The liblouis table filenames are mostly uniform so could in theory be used to identify most table languages and regions. Still, adding this explicitly to the table definitions sounds a whole lot safer. A stand-alone language-to-table map shouldn't be necessary, you can look up the appropriate table for a given key anyway. Something for another ticket I think.

Note that it's already possible for an add-on to get its directory from its own code: addonHandler.getCodeAddon().path. However, I guess we could also consider a utility function for an add-on to add a table.

Yeah, you could either have a single function to:

• Fix relative/absolute path quirks.
• Check if a table exists.
• Append the table to the list.
• Optionally start using the table immediately.
  ... Or you could let add-ons perform these steps themselves, which probably improves the re-usability of the code.

[nvaccessAuto]

Comment 17 by dkager on 2015-05-18 14:46
Revisiting this discussion, it appears there is one question I'm failing to answer:
Where to store the currently active translation/input tables?
The solutions, and their problems:

1. In braille.BrailleHandler. Problem: both braille.TextInfoRegion and brailleInput.BrailleInputHandler need to know BrailleHandler. This violates quite a few design patterns.
2. In the braille module itself, as (global) variables. This would work nicely but might conflict with NVDA's object-oriented coding style.
3. In a dedicated class TableHandler, either in its own module or in the braille module. Works, but has the same problem as (1): TextInfoRegion has to know this class somehow.

Long story short, the fix for allowing absolute paths for braille table filenames is easy enough, but it's a bit harder to find a good place for caching the currently active tables. You can't just read this from the config dictionary because this doesn't include absolute paths for stock tables, so it takes some extra computing.
There is one alternative: set the liblouis data directory at start-up and let it figure out if a path is absolute or not (it has logic to do this). This sounds to me like the cleanest solution. It also means we can continue to grab table filenames straight from the config dictionary.

[nvaccessAuto]

Comment 18 by dkager on 2015-06-14 11:10
Can this be blocked by #5137? Liblouis 2.6.3 has a function to check if a list of tables can be found and compiled. This should probably be done for all custom tables that are loaded, hence the dependency. It's also potentially useful to check the stock tables at build time as mentioned in comment:13.

[nvaccessAuto]

Comment 19 by dkager on 2015-07-23 19:18
Getting back to this...again. Here's a new version of this patch. It does the following, all discussed in earlier comments:

• Make braille.TABLES a list (called braille.tables) so add-ons can append to it.
• Detect if a table is one of the stock tables or a custom table. This is somewhat crude because it is done by checking whether the table path is absolute.
• Fall back to the US 8-dot computer braille table if a table cannot be found, both for custom and stock tables. This should avoid situations as described in Currently used Danish braille tables no longer exist #4986. The code uses the liblouis function lou_checkTable, added to the Python wrappers in v2.6.3. Thus this depends on Upgrade to liblouis 2.6.3 #5137.
• In the GUI, sort translation and input tables alphabetically.

All should be pretty straightforward and, if acceptable, should be all the custom braille table support NVDA needs IMO.

Branch: t3304