Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(oem/fv/ios): Dictionary Support #6197

Merged
merged 18 commits into from
Mar 29, 2022
Merged

Conversation

sgschantz
Copy link
Contributor

@sgschantz sgschantz commented Feb 3, 2022

Adds dictionary/lexical model support to FirstVoices iOS app.
Tapping a keyboard in the list causes transition to a detail page where the keyboard is displayed with other settings, including downloadable dictionaries, if available.

keyboard table

image

User Testing

The changes to the app are substantial and tests amount to a regression test

  • TEST_INSTALL: Verify that a FirstVoices keyboard is visible in iOS
  1. Go to iOS Settings -> General -> Keyboard -> Keyboards -> Add New Keyboard... and choose FirstVoices under the heading Third-Party Keyboards.
  2. Go to the Messages app
  3. Tap in the text edit field to begin typing a new message so that the system keyboard will appear
  4. Tap on the globe icon
  5. Verify that the keyboard has changed to 'EuroLatin (SIL)', available due to the FirstVoices install
  • TEST_TURN_ON_KEYBOARD: Verify that another FV keyboard can be added
  1. Open the FirstVoices app and tap the 'here' button on the main instructions page
  2. Tap the first keyboard in the list which will probably be 'Inuvialuktun' -- tap anywhere but on the information icon
  3. This should cause a transition to the settings page for Inuvialuktun
  4. Tap the uppermost switch to turn on this keyboard
  5. Tap the back arrow in the upper left arrow to return to the keyboards page
  6. On the keyboards page, verify that Inuvialuktun now has a checkmark next to it
  7. Switch to messages and verify that the Inuvialuktun keyboard is available from the globe menu
  • TEST_KEYBOARD_HELP: Verify that the help page is available
  1. For any keyboard listed in the keyboards page, tap the information icon
  2. Verify that a web page loads with helpful information about the keyboard that was selected
  • TEST_INSTALL_DICTIONARY: Verify that a dictionary can be installed and used
  1. On the keyboards page, scroll down and tap SENĆOŦEN under the BC Coast region heading
  2. Tap the switch to activate the SENĆOŦEN keyboard
  3. Verify that a switch is now available to activate the dictionary 'SENĆOŦEN (Saanich Dialect)'
  4. Tap the switch and choose Install on the resulting confirmation dialog
  5. Verify that related dictionary settings Suggest Predictions and Suggest Corrections have defaulted to the On position
  6. Open Messages and after switching to the SENĆOŦEN keyboard, verify that words are suggested while typing
    image
  • TEST_STATE_SAVED: Verify that the user's settings are stored safely between app invocations
  1. Make changes to various settings within the app, such as turning on certain keyboards and specifying different dictionary settings for the SENĆOŦEN keyboard
  2. Quit the app and restart it
  3. Verify that the previous settings are preserved
  • TEST_SWITCH_DEPENDENCIES: Verify that switches are only available as appropriate
  1. On the keyboards page, scroll down and tap SENĆOŦEN under the BC Coast region heading
  2. When the keyboard switch is in the Off position, verify that all other switches are Off and disabled (cannot be toggled)
  3. Tap the 'SENĆOŦEN keyboard switch so that it is in the On position and verify that the 'SENĆOŦEN (Saanich Dialect)' switch is now enabled
  4. Tap the 'SENĆOŦEN (Saanich Dialect)' switch so that it is now in the On position and verify (after dictionary download) that the Suggest Predictions and Suggest Corrections switches automatically default to the On position
  5. Tap the Suggest Predictions switch to the off position and verify that Suggest Corrections also moves to the off position where it is now disabled

@keymanapp-test-bot keymanapp-test-bot bot added the user-test-missing User tests have not yet been defined for the PR label Feb 3, 2022
@keymanapp-test-bot
Copy link

keymanapp-test-bot bot commented Feb 3, 2022

User Test Results

Test specification and instructions

  • TEST_INSTALL (PASSED): The "English - EuroLatin (SIL)" does appear following the instructions given. (notes)
  • TEST_TURN_ON_KEYBOARD (OPEN): retest
  • TEST_KEYBOARD_HELP (PASSED): The keyboard help page does show up. (notes)
  • TEST_INSTALL_DICTIONARY (OPEN): retest
  • 🟧 TEST_STATE_SAVED (BLOCKED): The FV keyboard doesn't show anything else beside "English - EuroLatin (SIL)". (notes)
  • 🟧 TEST_SWITCH_DEPENDENCIES (BLOCKED): The FV keyboard doesn't show anything else beside "English - EuroLatin (SIL)".
Results Template
# Test Results

* **TEST_TURN_ON_KEYBOARD (OPEN):** notes
* **TEST_INSTALL_DICTIONARY (OPEN):** notes

Test Artifacts

@darcywong00
Copy link
Contributor

UX question: Are the defaults on "Suggest Predictions" and "Suggest Corrections" off on Keyman for iPhone and iPad?

I think on Android, we default to "On" (less clicking after the user installs a lexical-model)

@sgschantz
Copy link
Contributor Author

In Keyman for iOS, I installed a new keyboard, and the switches for Predictions and Corrections were active and enabled by default. That was even with no dictionary installed.

That doesn't make much sense as those options should not be available until there is a dictionary, right? After a dictionary is installed, we can activate them and default them as enabled. Until then they should be disabled and inactive.

@sgschantz
Copy link
Contributor Author

A few notes on the details page:

  1. I labeled the switches Suggest Predictions and Suggest Corrections, using 'Suggest' instead of 'Enable' to try to be a little more descriptive. Switches always enable and disable things, so instead I wanted to describe what was being enabled or disabled. If there is a better word than 'suggest', let me know.
  2. After changing the enabling/graying out of controls, I think the Downloadable Dictionaries section should be moved up above the Suggest switches because the Suggest switches are dependent upon whether there is an active dictionary.
  3. Now we are calling the Suggest switches 'Language Settings' -- should they be Dictionary Settings instead? I think that would reinforce their purpose and make it more obvious why they are disabled when there is no dictionary.

The above changes differ from what we have in Keyman, but I think it would improve clarity of the UI.

As for how the state of the controls depend on each other, there is a pretty clear hierarchy:

  1. If the keyboard switch is off, then all other controls are off and disabled.
  2. If there is no dictionary downloaded or there is a dictionary and it is turned off, then both Suggest switches are off and disabled.
  3. If the Suggest Predictions switch is off, then the Suggest Corrections switch is off and disabled. (This is currently enforced in Keyman. Keyman also hides the switch, but that is not necessary.)

// try to read array
if !jsonArray.isEmpty {
let jsonModelMap = jsonArray.first
let name = jsonModelMap!["description"] as! String
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't name be?

Suggested change
let name = jsonModelMap!["description"] as! String
let name = jsonModelMap!["name"] as! String
Write Preview

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. I used the description field because it had the more succinct name:
"description": "SEN\u0106O\u0166EN (Saanich Dialect)",
vs
"name": "SEN\u0106O\u0166EN (Saanich Dialect) Lexical Model"

Not sure what I can expect from each field of the API, but this text goes in the switch name, so I need something brief. Is the lexical model definition incorrect?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should be using name. description is a long form which may be several lines long. It just so happens that the SNECOTEN model has a overly short description:

https://github.com/keymanapp/lexical-models/blob/fb306c0768caaf6f25cbb7abf9d7e4eb3ca74ef3/release/nrc/nrc.str.sencoten/nrc.str.sencoten.model_info#L3

vs name:

https://github.com/keymanapp/lexical-models/blob/fb306c0768caaf6f25cbb7abf9d7e4eb3ca74ef3/release/nrc/nrc.str.sencoten/source/nrc.str.sencoten.model.kps#L10

See for example https://api.keyman.com/model?q=bcp47:cmo-Khmr:

[
    {
        "license": "mit",
        "languages": [
            "cmo-khmr"
        ],
        "description": "Bunong Wordlist is based on the Bible wordlist of 1,157 unique entries with their frequencies.",
        "id": "sil.cmo.bw",
        "name": "Bunong Wordlist",
        "authorName": "[object Object]",
        "lastModifiedDate": "2022-02-18T18:41:02.203Z",
        "packageFilename": "https://keyman.com/go/package/download/model/sil.cmo.bw?version=1.1&update=1",
        "packageFileSize": 107384,
        "jsFilename": "https://downloads.keyman.com/models/sil.cmo.bw/1.1/sil.cmo.bw.model.js",
        "jsFileSize": 286398,
        "packageIncludes": [],
        "version": "1.1",
        "minKeymanVersion": "12.0",
        "sourcePath": "release/sil/sil.cmo.bw"
    }
]


class LexicalModelRepository {

private static let keymanLexicalModelApiUrl = "https://api.keyman.com/model?q=bcp47:"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@darcywong00
Copy link
Contributor

iOS is using an old script to download fv_all.kmp

# TODO: support passing -copy-keyboards, -debug, -clean etc in to build_keyboards
../common/build_keyboards.sh -download-keyboards

We can remove build_keyboards.sh if you include the following

. "$KEYMAN_ROOT/resources/build/build-download-resources.sh"

and then do

# Download default keyboard and dictionary packages
if [ "$DO_KEYBOARDS_DOWNLOAD" = true ]; then
echo "Copying keyboards.csv"
cp "$KEYMAN_ROOT/oem/firstvoices/keyboards.csv" "$KEYBOARDS_CSV_TARGET"
downloadKeyboardPackage "$KEYBOARD_PACKAGE_ID" "$KEYBOARDS_TARGET"
fi

@keymanapp-test-bot keymanapp-test-bot bot added has-user-test user-test-required User tests have not been completed and removed user-test-missing User tests have not yet been defined for the PR labels Mar 8, 2022
@mcdurdin
Copy link
Member

What should I do with any error information that I get back from KeymanEngine so that we can investigate this? Without Sentry in FV, I'm not sure where to send it. Is there any place that I can retrieve error text when Makara encounters an issue? I could pass it back and display in this new alert, but that doesn't seem too user friendly.

I think we aim to add Sentry to FV (I have created the fv-ios Sentry project at https://sentry.io/keyman/fv-ios/getting-started/apple-ios/). Otherwise, it is difficult, and I am certainly not keen on creating a new error-reporting mechanism for this.

If adding Sentry is too big a task for now, then we can temporarily add an alert while diagnosing these issues and then remove it again?

@darcywong00
Copy link
Contributor

Worth considering feeding the same changes to Keyman for Android

Yeah, we can consider that. I had a question on the SENCOTEN settings: "Keyboard Information" section. If there's multiple keyboards available for a language, which one would "Version" correspond to?

@sgschantz
Copy link
Contributor Author

sgschantz commented Mar 15, 2022

@MakaraSok I've added some code temporarily for troubleshooting the issue you are have activating the keyboard. This new version will display an alert with error information. Please retest and tell me what you see. I cannot reproduce this, so let me know if you have any more details that will isolate the issue.

@keymanapp-test-bot retest TEST_TURN_ON_KEYBOARD

@keymanapp-test-bot keymanapp-test-bot bot added the user-test-required User tests have not been completed label Mar 15, 2022
@sgschantz
Copy link
Contributor Author

@darcywong00 Good question on the version number. I think it's broken. We could display the version of the active dictionary if there is one, but that's kind of weak and still not obvious. I'm not sure what the best solution is.

@mcdurdin
Copy link
Member

Good question on the version number. I think it's broken.

This was obviously designed way back in the day with a one-keyboard-per-language pattern. Do we have any languages with more than one keyboard in the list? If not, then this is a brokenness that we can just ignore for now 😁 If we do, then I suggest for now just adding a version row for each keyboard and differentiating with the keyboard name in the caption (e.g. SENCOTEN version -> 9.1.1).

The detail is of course important but we also need to balance that with addressing more serious issues... so I am comfortable with a less than 100% elegant answer for this edge case.

@darcywong00
Copy link
Contributor

darcywong00 commented Mar 16, 2022

Do we have any languages with more than one keyboard in the list?

Looking at keyboards.csv, I see "ikt-Latn: Inuinnuaqtun (Latin)" is used by fv_inuvialuktun and fv_uummartmitutun.

But I'm thinking this should be fine because the FV apps separate Region --> Keyboard -->

so each keyboard has its own row...

@MakaraSok
Copy link
Collaborator

Test Results

  • TEST_TURN_ON_KEYBOARD (FAILED): The keyboard enabled and ticked, but it doesn't show up when switch to and fro from the iOS system keyboard. EuroLatin (SIL) keyboard is always there, not expected one which have previously been selected.
Screen.Recording.2022-03-17.at.9.26.47.AM.mov

@keymanapp-test-bot keymanapp-test-bot bot removed the user-test-required User tests have not been completed label Mar 17, 2022
@sgschantz
Copy link
Contributor Author

@mcdurdin So Makara is not seeing the alert that I added which implies that Keyman Engine is not throwing an error and it looks like a success to FV. Do you know of any other way to troubleshoot this in Makara's environment?
@MakaraSok can you describe what environment you're running this in or provide any more info that might help me reproduce it?

@mcdurdin
Copy link
Member

@mcdurdin So Makara is not seeing the alert that I added which implies that Keyman Engine is not throwing an error and it looks like a success to FV. Do you know of any other way to troubleshoot this in Makara's environment?

I guess running Console on his machine and looking at logs from the simulator may provide some enlightenment?

@sgschantz
Copy link
Contributor Author

@MakaraSok @darcywong00 @mcdurdin I just realized that this problem Makara is seeing is the same as #6226 which appeared with the old FV iOS code. The change of Darcy's should not have broken the iOS FV app, should it? Or maybe that's why there was an iOS test case on #6226?

@mcdurdin
Copy link
Member

I just realized that this problem Makara is seeing is the same as #6226 which appeared with the old FV iOS code. The change of Darcy's should not have broken the iOS FV app, should it? Or maybe that's why there was an iOS test case on #6226?

Yes, I think that this problem is not new to this PR and is specific to the simulator.

@mcdurdin mcdurdin modified the milestones: B15S4, B15S5 Mar 21, 2022
@MakaraSok
Copy link
Collaborator

Retest all failed/blocked

I've retested this again today and it doesn't work as expected like what was experienced prior. Note that the test was done on a simulator with its "all content and settings" erased being installing the .

  • TEST_TURN_ON_KEYBOARD: FAILED Please see the screen recording below.
FV.doesn.t.show.as.sys.kb.480p.mov
  • TEST_INSTALL_DICTIONARY: FAILED The very attempt to install the dictionary disable itself after clicking on the "install" button. Please see the screen recording below.
unable.to.install.dictionary.mov

The second attempt to enable the dictionary got through without bouncing back, but it still doesn't show up when summoned. Please see the screen recording below.

no.kb.shown.no.dictionary.visible.mov

The blocked ones stay "Blocked".

@MakaraSok
Copy link
Collaborator

@sgschantz
Copy link
Contributor Author

sgschantz commented Mar 28, 2022

Ready for retest, but need to have a way to either test a release version of FV on the simulator or a real device. Debug FV on the simulator works well from Xcode. @mcdurdin any ideas on how to change the relase build so that it will behave the same on simulator and device?

@keymanapp-test-bot retest TEST_TURN_ON_KEYBOARD TEST_INSTALL_DICTIONARY

@keymanapp-test-bot keymanapp-test-bot bot added the user-test-required User tests have not been completed label Mar 28, 2022
@mcdurdin
Copy link
Member

ny ideas on how to change the relase build so that it will behave the same on simulator and device?

It's really hard. The simulator does not appear to behave identically to devices in all circumstances -- particularly that boundary between apps and app extensions seems different.

The best solution may be to merge to beta and do a beta release build which is then available in TestFlight. Otherwise, unless you are in the same room (where you can plug the device into your computer and do a build and run from XCode), I don't know how it can be done.

@sgschantz sgschantz merged commit a67d082 into beta Mar 29, 2022
@sgschantz sgschantz deleted the feat/oem/fv/ios/dictionary-support branch March 29, 2022 01:02
@keyman-server
Copy link
Collaborator

Changes in this pull request will be available for download in Keyman version 15.0.224-beta

@MakaraSok
Copy link
Collaborator

MakaraSok commented Mar 30, 2022

Test Results

Tested on a physical device, iPad mini 6 running iOS 15.4.

  • TEST_TURN_ON_KEYBOARD (FAILED): Right after a keyboard was selected/installed, it doesn't appear in the keyboard menu unless restarting the device.
  • TEST_INSTALL_DICTIONARY (FAILED): The same behavior mentioned above.

Note that when removing a keyboard from the FV app, every keyboard was removed with it leaving only EuroLatin keyboard to choose from.

@MakaraSok
Copy link
Collaborator

Test Results

  • TEST_TURN_ON_KEYBOARD (PASSED): no issue found
  • TEST_INSTALL_DICTIONARY (PASSED): no issue found
  • TEST_STATE_SAVED (PASSED): no issue found
  • TEST_SWITCH_DEPENDENCIES (PASSED): no issue found

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

None yet

5 participants