Help creating small english dictionary

[Charles7z]

I only have access to android devices, phone & older tablet, and I don't know programming... At least not what the dictionary creation talks about.

What I'm wanting to do is create a smaller english dictionary. I think it'll help with suggestions and autocorrect, and overall speed for tap typing and gesture, maybe.

I've got the 50k word list from the create dictionary link here. I'll go through it and remove words that are not needed, wrong, etc. IF there's anyone else willing to help me finish it by running the necessary compiling? I don't want to start the editing if it's a dead end. Thanks!

Anyone? 😁😁😁

[Helium314]

For compiling you just need to run java -jar dicttool_aosp.jar makedict -s <language>_wordlist.combined -d main_<language>.dict.
Do you have the wordlist file finished already?

[Helium314]

smalltest.dict.zip

I'm not sure but HeliBoard can load multiple dictionaries under the same language?

Yes, the dictionaries just need a different type (replace main with medical, emoji, tech, ...).
I didn't test how many dictionaries can be loaded, but I assume there is no limit you can reach accidentally. Only thing to look out is that there might be performance issues when you have a lot of dictionaries (not sure though).

what they need vs performance

Do you have performance issues? I'm using an ancient Galaxy S4 Mini, and the only case when it's noticeably slow is when using a debug build.

Another question for you, doesn't HeliBoard also do it's own "bigram" for generating next word suggestions based on use? So using bigram in the dictionary is really more of a head start but is over ridden by the algorithm?

Yes, the bigrams in the dictionary are just a head start.

[Charles7z]

@Helium314

I'm plugging away, mainly in circles 🤦‍♂️

I've got a quick question

dictionary=main

I'm uncertain what can be used other than "main".

Performance — I was referring to typing efficiency/performance not hardware however, the keyboard hangs and crashes at times when many large copies of text have been done and then you quick tap the clipboard.

Edit
Forgot to ask
Can phrases be used within word= and/or birgram?

[Helium314]

I'm uncertain what can be used other than "main".

You can simply test what works and what doesn't. If you want to be safe, you should stick to short words in lowercase ascii. But you can try anything and check what happens. That would probably be good information to add to https://codeberg.org/Helium314/aosp-dictionaries#wordlist-information
As far as I know, main is the only value that is treated in some sort of special way.

Can phrases be used within word= and/or birgram?

Probably they can be used, but might end up in a later position in the suggestions list, so you may have to open the more suggestions view to actually find them.
Further, you are limited to 48 characters as far as I remember (this is a limit in the app, and possibly also in dicttool_aosp).

[Charles7z]

I've noticed that words with f=0 are gestured no problem when block offensive words is enabled and show in the extra suggestions.

For gesture typing f= seems almost irrelevant. And explains why gesture typing has been very off for me.

I've tried this in the standard US and no language with the experimental dictionary. Enabling block offensive words is meaningless with gesture typing. I guess that has to do with importing the library?

Things work with tap typing as expected.

[Charles7z]

@Helium314

I've been researching several things with dictionaries and have found some things that will help HeliBoard and other keyboards, significantly.

But I need your help to run my last tests.

I'm doing this all on a pixel, no computer, so if you could do a couple things to help me finish this work — I promise it will be worth while — it will be greatly appreciated.

1. If you would please compile these word lists to dictionary. Zip uploaded.
2. I need an APK of HeliBoard with no dictionaries in it. There's issues with using the no language setting, and I want to make sure there's no conflicts with the glide library and any other languages, so that I'll have clear results.

Thanks!!!

charles7z_dict_files.zip

[MXC48]

Can I use the data from http://www.lexique.org to create a dictionary?

[Helium314]

Generating word lists from this might need some processing, but should work. The license should be fine too.

[MXC48]

Generating word lists from this might need some processing, but should work. The license should be fine too.

What kind of treatment should be used? Do we need to retain the words and their frequency?

[Helium314]

Yes, you need the word and a frequency, see e.g. https://codeberg.org/Helium314/aosp-dictionaries/src/branch/main/wordlists/sample.combined
The actual f value in the word list should be the logarithm of the actual frequency, and scaled so all words are between 15 (I think) and 255.