Add bip39 Indonesian wordlist #621

perlancar · 2018-01-01T03:26:41Z

How the wordlist is produced:

Download and uncompress https://dumps.wikimedia.org/idwiki/latest/idwiki-latest-pages-articles.xml.bz2 (the version used when producing the wordlist is 2017-121-30).
Count the words in all the articles inside articles.xml using this script https://github.com/perlancar/perl-WordLists-ID-Common/blob/master/devscripts/count-words-in-mediawiki-articles . The result is
https://raw.githubusercontent.com/perlancar/perl-WordLists-ID-Common/master/devdata/words.txt .
Curate the words manually (mostly removing non-Indonesian words). The result is https://raw.githubusercontent.com/perlancar/perl-WordLists-ID-Common/master/devdata/words-curated.txt . You can diff this two wordlist text file to see the difference.
Generate the BIP39 Indonesian wordlist using this script https://github.com/perlancar/perl-WordList-ID-BIP39/blob/master/devscripts/gen-wordlist . This script basically picks the most frequent words in words-curated.txt that are not already in the English, Spanish, French, and Italian BIP39 wordlist.

dabura667 · 2018-01-01T06:38:00Z

Technical Checklist ACK 2b35f48

Checked:

Is NFKD normalized list
No BOM
Has single trailing LF line break and is separated by LF line breaks

perlancar · 2018-01-01T07:13:44Z

@dabura667 Like English, Indonesian only uses 26 Latin letters and encodable in ASCII so no need to put in BOM and asciibetical sorting suffices. Please advise about trailing LF and LF line breaks, because that's how most of the other wordlist files use too.

dabura667 · 2018-01-01T08:07:19Z

ACK means "ok to merge" and "Technical Checklist" means I checked for formatting errors and everything is OK.

perlancar · 2018-01-01T08:13:34Z

Ah ok, thanks.

As an undocumented test feature, this adds the proposed Indonesian wordlist from bitcoin/bips#621 . The wordlist is subject to change or removal, pending resolution of the BIP pull request. Do NOT use this for any purpose other than testing, unless or until the wordlist be finalized and accepted in final form. SHA-256 for indonesian.txt in its current version: a88f63a3e2387453d0c6de09ffe06d318968fe76052c6324169f6a41fa2247e0 The Wikileaks wlupld3ptjvsgwqw.onion, encoded in proposed Indonesian: perjalanan fokus jujur suaminya otak masuk kamu roket

nym-zone · 2018-01-05T15:18:31Z

Has this any independent review from other native speakers or experts in the language? The earlier BIP wordlist addition pull requests witnessed much lively discussion and examination. One was superseded by a new proposal after significant problems were found.

Since this is said to be ASCII, it is easy to check some basic characteristics in addition to those checked by @dabura667:

$ grep '^[^a-z]' indonesian.txt
$ grep -Eo '^[a-z]{0,3}$' indonesian.txt
$ grep -Eo '^[a-z]{4}' indonesian.txt | sort -s | uniq | wc -l
2048

Sample 24-word mnemonic (outside of code-tagging to permit line breaks):

$ easyseed -b 256 -l id
keuskupan utuh kegunaan serta pesisir mungkin reguler cermin langsung enam parkir lari gaib bensin babak dinilai meluncurkan mandiri bijaksana keamanan domestik bercerita prefektur legislatif

@perlancar, a question from an implementer: Are these strings for identifying this wordlist sensible and appropriate for Indonesian-speaking users?

+	LANG(indonesian,	u8"Bahasa Indonesia",	"id",	ascii_space ),

I have proposed on bitcoin-dev that native language strings and short ASCII codes should be standardized:
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-January/015498.html

nym-zone · 2018-01-05T15:37:50Z

@perlancar, I recommend that you split 4fcfed7, “Add Perl bip39 implementation: Bitcoin::BIP39”, into a separate pull request. That should be reviewed separately from the wordlist; and it’s not even mentioned in the title or description of this pull request #621. People who glance through the list of pull requests will not even realize that this exists.

It is good that you did atomic commits, so that 2b35f48 (the proposed Indonesian wordlist) and 4fcfed7 (listing of the Perl module in bip-0039.mediawiki itself) can be handled separately.

I am not a maintainer here; therefore, I can only make a “recommendation”.

perlancar · 2018-01-05T23:15:43Z

@nym-zone I did separate into two pull requests, but admittedly I didn't create a branch for the first PR. And then when I created the second PR, Github merged the two.

perlancar · 2018-01-05T23:23:24Z

@nym-zone Thanks for the input. I did publish the proposed Indonesian wordlist as a Perl module (mentioned above) in hope of gathering some input, but of course the number of Indonesian Perl users is negligible. I will try to solicit input from more communities. For your information, I am a native Indonesian speaker. The wordlist were selected from most common Indonesian words which I have curated manually so I would say they are sensible. But that is my opinion.

Buglet: A wordlist not yet accepted into the BIP repository should not be visible in the user interface. It is included only for purposes of testing, since it may be changed or deleted at any time. The -L option automatically listed *all* available wordlist languages; now it has a special-case check to not list Indonesian. Change this back if/when BIP PR bitcoin/bips#621 be accepted.

Access to proposed wordlists not (yet) accepted into the BIP repository is now controlled by the -W option. This option is and will remain undocumented in the manpage and other user documentation. The purpose of this undocumented flag is to permit testing proposed wordlists and generating test vectors, without inducing users to rely on wordlists which may be changed or removed at any time without notice. Users MUST NOT generate actual wallets based on proposed wordlists; so doing could result in unrecoverable wallets and permanent funds loss. I am now ready to import three more proposed wordlists I had not hereto seen amongst BIP pull requests: - Russian, bitcoin/bips#432 - Ukrainian, bitcoin/bips#442 - Czech, bitcoin/bips#493 Wordlist already imported for testing, now hidden behind -W option: - Indonesian, bitcoin/bips#621 (b2f66ba, special-cased d03ddae) Further changes hereby made, with due apologies for a non-atomic commit: - Integrating the -W option necessitated a general cleanup and overhaul of the options-handling code. - Whilst overhauling the options, I noticed that the documented -P option functionality was broken/nonexistent. Fixed.

These are generated with easyseed and the bip39_vectorgen.sh script from 5f35cd0. There are vectors in twelve languages: The eight in the BIP repository, and four more with pending proposals. Three of the vectors for proposed languages are for wordlists which I have modified: - Russian (234c66c, bitcoin/bips#432) - Ukrainian (08a05b4, bitcoin/bips#442) - Czech (ba25dfa, bitcoin/bips#493) The wordlist for Indonesian (bitcoin/bips#621) is unmodified from the proposal. Ironically, easyseed does not yet self-test itself with these. That will be added in a future release, to verify consistency between builds. For now, I publish these to aid in interoperability testing between implementations.

ubunteroz · 2018-05-27T15:59:03Z

Hello, native Indonesian speaker here. Thanks @perlancar for your effort!
Quick review from your wordlist:

It's better to remove conjuctions (such as atau, tetapi, yaitu, and yakni) to avoid confusion.
Also, non-root words can cause another confusion (peperangan -> perang, pedesaan -> desa, menyebabkan -> sebab).
Limiting wordlist to max. 8/9-letter words can help us to remember easily.

perlancar · 2018-06-02T21:42:08Z

Hi Surya (@ubunteroz), thanks for the comments. All are good. I will manually construct an edited new wordlist when I have some free time, but PR's or patches are welcome :)

luke-jr · 2018-07-05T05:19:47Z

@dabura667 Only authors are supposed to ACK changes to BIPs.

In this case, these people: @slush0 @prusnak @voisine @ebfull

heri16 · 2019-07-03T03:55:06Z

bip-0039/indonesian.txt

+upacara
+upah
+upaya
+upeti


This word is unfamiliar even to native speakers.

Sorry, by "this word" which one are you referring to? Because GitHub Web UI is showing 4 words: upacara, upah, upaya, upeti. I would argue that they are familiar to native speakers. What is your basis to say that they are not? Let's take "upeti" for example, this word is found 792 times in the Wikipedia article (see PR description which I have just updated to describe the process of producing the wordlist).

Native indonesian speaker here. Instead of the Wikipedia Corpus (which contain too many technical terminology), have we considered the "Sari Kata Bahasa Indonesia" book?

This book is used nationwide in junior/primary schools. It would ensure anyone with a basic education would be able to understand our wordlist.

This same book is also used as a reference to test if a foreigner can be permitted to work in Indonesian companies.

This is a good idea, but what about license? And is there an online version for it?

heri16 · 2019-07-03T03:56:55Z

bip-0039/indonesian.txt

+tunduk
+tunggal
+tuntutan
+turbin


This word is unfamiliar even to native speakers.

Which word are you referring to? tunduk, tunggal, tuntutan, turbin? All of them? What is the basis? (See my comment on the upeti word above).

heri16 · 2019-07-03T03:57:29Z

bip-0039/indonesian.txt

+tragedi
+trailer
+transportasi
+trek


This word is unfamiliar even to native speakers.

Which word are you referring to? trek? I tend to agree.

heri16 · 2019-07-03T03:58:07Z

bip-0039/indonesian.txt

+tikus
+timah
+timbul
+timnya


This word is a conjugation.

heri16 · 2019-07-03T03:58:40Z

bip-0039/indonesian.txt

+tersebut
+tertentu
+terutama
+terwujud


This word is a conjugation.

I agree that conjugated words should be avoided.

heri16 · 2019-07-03T04:03:31Z

bip-0039/indonesian.txt

+simbol
+sinar
+sinetron
+singel


This imported word is has multiple spellings debated by native speakers.

heri16 · 2019-07-03T04:05:53Z

bip-0039/indonesian.txt

+sikap
+siklus
+silat
+silinder


Technical term for most native speakers.

I can agree to this.

heri16 · 2019-07-03T04:07:11Z

bip-0039/indonesian.txt

+senjata
+sensus
+sentral
+senyawa


Is a Conjucation or Technical term for most native speakers.

heri16 · 2019-07-03T04:09:59Z

bip-0039/indonesian.txt

+semakin
+sembilan
+sementara
+seminggu


Conjugation

heri16 · 2019-07-03T04:11:10Z

bip-0039/indonesian.txt

+sama
+sambil
+sampai
+samurai


Imported word not familiar to most native speakers.

Agreed on imported words, though I don't share your opinion of "samurai" being not familiar with most native speakers.

heri16 · 2019-07-03T04:47:04Z

Native indonesian speaker here. Instead of the Wikipedia Corpus (which contain too many technical terminology), have we considered the "Sari Kata Bahasa Indonesia" book?

This book is used nationwide in junior/primary schools. It would ensure anyone with a basic education would be able to understand our wordlist.

This same book is also used as a reference to test if a foreigner can be permitted to work in Indonesian companies.

perlancar · 2019-07-03T15:33:07Z

I have created another wordlist. The source is still from Wikipedia Indonesia, but this time I manually curate the BIP wordlist using this criteria:

Words from 4 to 10 letters long.
Avoid conjugated words, choose only root words.
Avoid prepositions, conjunctions, pronouns (e.g. dan, atau, jika, kamu, saya, ...).
Avoid technical words when possible.
Avoid loan words when possible.
Avoid words that have multiple competing spellings, if possible.
The above criteria is balanced/countered by this: I want to make the BIP wordlist unique by its 4-letters. So even if user does not type or mistype some letters at the end of words, they can still be corrected.

The resulting BIP wordlist is put here: https://github.com/perlancar/perl-WordList-ID-BIP39/blob/master/devdata/words-bip.txt. I haven't made a change to this PR. Inputs/comments welcome.

As you might see, some of the words are indeed not very popular to native speakers, because I also want to satisfy criteria number 7.

Also useful is the larger wordlist from which I curated this BIP wordlist: https://github.com/perlancar/perl-WordList-ID-BIP39/blob/master/devdata/words.txt . I indented the words which I want to use in the BIP wordlist. You can suggest corrections or changes by submitting a PR which changes this file.

luke-jr · 2021-07-02T21:29:56Z

For now, the author(s) of BIP 39 have decided not to accept any further word lists into BIP 39 itself, and encourage adding new ones to the WLIPs repo here: https://github.com/p2w34/wlips

Add bip39 Indonesian wordlist

2b35f48

Add Perl bip39 implementation: Bitcoin::BIP39

4fcfed7

luke-jr added the Proposed BIP modification label Jul 5, 2018

dlwhitehurst mentioned this pull request Jul 11, 2018

Add Indonesian Word List once Accepted (bip39) ciwise/bip39c#13

Open

DonaldTsang mentioned this pull request Dec 24, 2018

Binary Lists grempe/diceware#44

Closed

22 tasks

heri16 suggested changes Jul 3, 2019

View reviewed changes

luke-jr closed this Jul 2, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add bip39 Indonesian wordlist #621

Add bip39 Indonesian wordlist #621

perlancar commented Jan 1, 2018 •

edited

Loading

dabura667 commented Jan 1, 2018

perlancar commented Jan 1, 2018

dabura667 commented Jan 1, 2018

perlancar commented Jan 1, 2018

nym-zone commented Jan 5, 2018 •

edited

Loading

nym-zone commented Jan 5, 2018

perlancar commented Jan 5, 2018 •

edited

Loading

perlancar commented Jan 5, 2018

ubunteroz commented May 27, 2018

perlancar commented Jun 2, 2018

luke-jr commented Jul 5, 2018

heri16 Jul 3, 2019

perlancar Jul 3, 2019

perlancar Jul 3, 2019

heri16 Jul 3, 2019

perlancar Jul 3, 2019

heri16 Jul 3, 2019

perlancar Jul 3, 2019

heri16 Jul 3, 2019

heri16 Jul 3, 2019

perlancar Jul 3, 2019

heri16 Jul 3, 2019

perlancar Jul 3, 2019

heri16 Jul 3, 2019

perlancar Jul 3, 2019

heri16 Jul 3, 2019

perlancar Jul 3, 2019

heri16 Jul 3, 2019

perlancar Jul 3, 2019

heri16 Jul 3, 2019

perlancar Jul 3, 2019

heri16 commented Jul 3, 2019 •

edited

Loading

perlancar commented Jul 3, 2019 •

edited

Loading

luke-jr commented Jul 2, 2021

Add bip39 Indonesian wordlist #621

Add bip39 Indonesian wordlist #621

Conversation

perlancar commented Jan 1, 2018 • edited Loading

dabura667 commented Jan 1, 2018

perlancar commented Jan 1, 2018

dabura667 commented Jan 1, 2018

perlancar commented Jan 1, 2018

nym-zone commented Jan 5, 2018 • edited Loading

nym-zone commented Jan 5, 2018

perlancar commented Jan 5, 2018 • edited Loading

perlancar commented Jan 5, 2018

ubunteroz commented May 27, 2018

perlancar commented Jun 2, 2018

luke-jr commented Jul 5, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

heri16 commented Jul 3, 2019 • edited Loading

perlancar commented Jul 3, 2019 • edited Loading

luke-jr commented Jul 2, 2021

perlancar commented Jan 1, 2018 •

edited

Loading

nym-zone commented Jan 5, 2018 •

edited

Loading

perlancar commented Jan 5, 2018 •

edited

Loading

heri16 commented Jul 3, 2019 •

edited

Loading

perlancar commented Jul 3, 2019 •

edited

Loading