Crash while loading BHFST file

[ftyers]

The bhfst works with divvunspell:

05-07 13:14:12.558  3861  5596 I ActivityManager: Start proc 6056:org.nlhowell.ddo/u0a298 for service org.nlhowell.ddo/com.android.inputmethod.latin.LatinIME
05-07 13:14:13.516  6056  6056 D onCreate: Locale: ddo
05-07 13:14:13.518  6056  6056 W System.err: java.io.FileNotFoundException: layouts/ddo.json
05-07 13:14:13.608  6056  6056 D DivvunUtils: getSpeller() for ddo
05-07 13:14:13.760  6056  6056 F libc    : Fatal signal 6 (SIGABRT), code -6 in tid 6056 (rg.nlhowell.ddo)
05-07 13:14:13.893  6208  6208 F DEBUG   : pid: 6056, tid: 6056, name: rg.nlhowell.ddo  >>> org.nlhowell.ddo <<<
05-07 13:14:13.904  6208  6208 F DEBUG   :     #02 pc 00000000000b2538  /data/app/org.nlhowell.ddo-ynF5hEComh1mmiLOUVspPQ==/lib/arm64/libdivvunspell.so (_ZN3std3sys4unix14abort_internal17he54cec259ec454baE+4)
05-07 13:14:13.904  6208  6208 F DEBUG   :     #03 pc 00000000000bac30  /data/app/org.nlhowell.ddo-ynF5hEComh1mmiLOUVspPQ==/lib/arm64/libdivvunspell.so (rust_oom+36)
05-07 13:14:13.904  6208  6208 F DEBUG   :     #04 pc 00000000000cf514  /data/app/org.nlhowell.ddo-ynF5hEComh1mmiLOUVspPQ==/lib/arm64/libdivvunspell.so (_ZN5alloc5alloc18handle_alloc_error17h3272256472896dcbE+4)
05-07 13:14:13.904  6208  6208 F DEBUG   :     #05 pc 00000000000b43a8  /data/app/org.nlhowell.ddo-ynF5hEComh1mmiLOUVspPQ==/lib/arm64/libdivvunspell.so (_ZN3std10sys_common12os_str_bytes5Slice8to_owned17h09902bd5dac085a4E+52)
05-07 13:14:13.904  6208  6208 F DEBUG   :     #06 pc 00000000000b39ec  /data/app/org.nlhowell.ddo-ynF5hEComh1mmiLOUVspPQ==/lib/arm64/libdivvunspell.so (_ZN3std4path4Path11to_path_buf17he4f43a16748e5dc6E+16)
05-07 13:14:13.904  6208  6208 F DEBUG   :     #07 pc 0000000000085df4  /data/app/org.nlhowell.ddo-ynF5hEComh1mmiLOUVspPQ==/lib/arm64/libdivvunspell.so (divvun_thfst_chunked_box_speller_archive_open+60)
05-07 13:14:13.904  6208  6208 F DEBUG   :     #08 pc 000000000000fe70  /data/app/org.nlhowell.ddo-ynF5hEComh1mmiLOUVspPQ==/lib/arm64/libjnidispatch.so (ffi_call_SYSV+96)
05-07 13:14:13.904  6208  6208 F DEBUG   :     #09 pc 000000000000f660  /data/app/org.nlhowell.ddo-ynF5hEComh1mmiLOUVspPQ==/lib/arm64/libjnidispatch.so (ffi_call+292)
05-07 13:14:13.904  6208  6208 F DEBUG   :     #10 pc 0000000000006cf0  /data/app/org.nlhowell.ddo-ynF5hEComh1mmiLOUVspPQ==/lib/arm64/libjnidispatch.so
05-07 13:14:13.904  6208  6208 F DEBUG   :     #11 pc 000000000000fbac  /data/app/org.nlhowell.ddo-ynF5hEComh1mmiLOUVspPQ==/lib/arm64/libjnidispatch.so
05-07 13:14:13.904  6208  6208 F DEBUG   :     #12 pc 000000000000fefc  /data/app/org.nlhowell.ddo-ynF5hEComh1mmiLOUVspPQ==/lib/arm64/libjnidispatch.so
05-07 13:14:13.904  6208  6208 F DEBUG   :     #13 pc 0000000000025e18  /data/app/org.nlhowell.ddo-ynF5hEComh1mmiLOUVspPQ==/oat/arm64/base.odex (offset 0x20000)
05-07 13:14:14.951  3861  4768 I ActivityManager: Process org.nlhowell.ddo (pid 6056) has died: fore IMPB

Here is the bhfst file base-64 encoded:

ddo.bhfst.b64.txt

You can decode it using :

$ base64 -d < filename.txt > ddo.bhfst

[nlhowell]

I'm the reporter; maybe this will subscribe me?

[bbqsrc]

The crash is happening because parsing the path fails, which is rather strange. It's very probably a bug in the foreign-function interface to DivvunSpell.

What path is being passed from the Java/Kotlin side of things?

[snomos]

I'm the reporter; maybe this will subscribe me?

Test: can @nlhowell see this answer?

[nlhowell]

Can you tell me how to figure this out? This is created from a kbdgen project, so I don't interact much with the Java side

[nlhowell]

Here's the line from the kbdgen build: [2020-05-07T11:50:56Z INFO kbdgen.gen.android] Adding 'ddo.bhfst' to 'app/src/main/assets/dicts'…

[nlhowell]

And grepping the giella-ime output from kbdgen for bhfst, I find:

./app/build/intermediates/incremental/mergeDebugAssets/merger.xml:</source

[bbqsrc]

@nlhowell @ftyers what brand and model of device where you testing this on?

[nlhowell]

I'm%20using%20an%20LG%20G6%20(h870ds).%0A%0AAre%20.bhfst%20spellers%20working%20for%20you%3F%20Can%20you%20provide%20an%20APK%20with%20a%20working%0Abhfst%20speller%3F%20If%20it%20was%20generated%20from%20kbdgen%2C%20can%20you%20provide%20a%20working%0Akbdgen%20bundle%3F%20If%20you%20think%20it's%20my%20device%2C%20I%20would%20be%20happy%20to%20test%20these%20to%0Asee%20if%20we%20can%20pin%20down%20the%20problem.%0A%0AI%20see%20there%20are%20some%20new%20commits%20on%20%60develop%60%3B%20I'll%20try%20testing%20again...%0A%0A%0A%0A

[nlhowell]

I'm using an LG G6 (h870ds).

Are .bhfst spellers working for you? Can you provide an APK with a working
bhfst speller? If it was generated from kbdgen, can you provide a working
kbdgen bundle? If you think it's my device, I would be happy to test these to
see if we can pin down the problem.

I see there are some new commits on develop; I'll try testing again...

Oops, double-encoded; sorry.

[bbqsrc]

Was just checking a theory that it might be a 32-bit vs 64-bit issue, but this phone is 64-bit so probably not that.

I'll be looking into this issue tomorrow.

[bbqsrc]

I haven't forgotten about this, other priorities had taken hold. I will be looking at this ~Wednesday.

[nlhowell]

No worries! Thanks for keeping me informed :)

[bbqsrc]

I have found some potential issues with string handling of errors in the DivvunSpell bindings for Android (the length was never added as part of the callback, which might have caused some very strange side effects).

I am unsure if this is related to your issue, but I'll have fixed that one soon.

[nlhowell]

Well, I tried a couple of things:
(1) Running again kbdgen (fetching the latest giellakbd-android/develop)
no change

(3) Updating to the latest version of kbdgen
removed bhfst speller support?

It seems bhfst speller support was removed in
cd4aeebc01893b6c9f43e17a6c0e80cc4bf7ff84, is that right? Or do I need to adjust
my kbdgen project somehow? I tried looking at the patches and didn't see
anything suggestive.

Cheers, and thanks for your work on this,
Nick

[nlhowell]

Any progress? Is there anything I can do to help?

Cheers,
Nick

[bbqsrc]

@nlhowell unfortunately, not yet, and today is the midsummer public holiday in Sweden so I'm not even working. 😄

What I can tell you is, next week embedding BHFST files without needing to use our CI services will return to kbdgen. I have to debug some other Android issues at the same time, so I hope that the issue you're having will become evident.

[nlhowell]

OK, no problem! Thanks for the quick response, and have a pleasant holiday!

[nlhowell]

Any update on this?

Cheers,
Nick

[bbqsrc]

@nlhowell oh wow, sorry for dropping the ball on this yet again. I'm on vacation now, but I'm re-adding the ability to add BHFST files to the Android build to kbdgen right now. (Ironically, I actually have time now, heh).

I worry that changes to the Android codebase might make it ignore local BHFST files however, so that might be a bit of an issue, but one thing at a time I suppose.

[bbqsrc]

Hmm, indeed, the current version of the Android keyboard effectively requires that the BHFST files is a Pahkat package found in a specific directory within the app's data directory. (I have not been working on either of these codebases directly lately, so this is actually a bit of a surprise to me.)

I will have a quick look tomorrow, as it shouldn't be hard to revert some of the logic and make this work for you. :)

Also I mean tomorrow this time, nobody else is around to shift my priorities hahaha

[bbqsrc]

I would note @ftyers, that it might be worth discussing a support contract in the future if priority support becomes a need. As much as I've tried to make this a priority, without funding, resource availability is a bit of a challenge. If you're satisfied with the delays so far, no problem, otherwise, shoot me an email and we can discuss. 😄

[ftyers]

@bbqsrc this isn't my issue, I just filed it on behalf of @nlhowell because he struggles with accessibility issues of Javascript-heavy web interfaces.

[bbqsrc]

@ftyers fair enough. I was under the impression that this was related to a project at Indiana University that you are supervising or otherwise responsible for. Sorry if I got my wires crossed. 🙂

[bbqsrc]

I have confirmed the linked ddo.bhfst file works with the most recent develop branch of this repository and the develop branch of kbdgen. Try it out.