grep: Invalid collation character

[freebiesoft]

When I run the clean-lex.sh file I get grep: Invalid collation character error

[polm]

Must be an issue with your locale. Try calling export LC_COLLATE=C or export LC_COLLATE=C.UTF-8 first, that should fix it.

If that doesn't fix it, let me know. If that does fix it, I can add it to the script.

[freebiesoft]

@polm I worked around it by substituting the grep string's special characters (i.e. "wide latin" characters) with their respective unicode hex numbers i.e. '^([A-Za-z\x{FF21}-\x{FF3A}\x{FF41}-\x{FF5A}]|[0-9\x{FF10}-\x{FF19}]*),', although it does confuse me a little as to why grep -E '^([A-Za-z\x{FF21}-\x{FF3A}\x{FF41}-\x{FF5A}]|[0-9\x{FF10}-\x{FF19}]*),' lex.csv | wc -l only outputs just over 100, whereas you claim (in the clean-lex.sh code comments) that the script should remove 232 entries... however, I am using the latest version of unidic.

[polm]

Well that's weird. Can you tell me what your LOCALE and OS are and confirm whether or not the LC_COLLATE setting fixes things for you?

The number of entries I mentioned is for UniDic 2.3.0, which is still the latest version. So if you're just getting 100 it's possible something is going wrong.

[freebiesoft]

@polm I just realised I have downloaded the spoken japanese dict (as that is version 3.0.1.1). I'll download the written language one (tomorrow) and check again :)

[polm]

@freebiesoft Any update on this?

[polm]

Closing due to lack of activity.

[srctaha]

@polm FYI

• UniDic: 2.3.0
• OS: macOS Big Sur 11.2.1
• Locale:

  LANG=""
  LC_COLLATE="C"
  LC_CTYPE="UTF-8"
  LC_MESSAGES="C"
  LC_MONETARY="C"
  LC_NUMERIC="C"
  LC_TIME="C"
  LC_ALL=
  

With grep (BSD grep) 2.5.1-FreeBSD, grep -E '^([A-Za-zＡ-Ｚａ-ｚ]|[0-9０-９]*),' lex.csv | wc -l returns 863869, removing 8962 entries.

With ggrep (GNU grep) 3.6, ggrep -P '^([A-Za-z\x{FF21}-\x{FF3A}\x{FF41}-\x{FF5A}]|[0-9\x{FF10}-\x{FF19}]*),' lex.csv | wc -l returns 232.

[polm]

Thanks for the report. I'll add the collate setting to the script to avoid issues like this.

[polm]

Oh wait... your locale settings look right and it's still not doing what I would expect. Hm.

I'll take a look at this later, but I would recommend using ggrep.

[srctaha]

Sorry for misleading. Mine is indeed not a locale issue. ggrep -E '^([A-Za-zＡ-Ｚａ-ｚ]|[0-9０-９]*),' lex.csv | wc -l returns 232, as expected.

[polm]

So I'm looking into the difference between BSD and GNU grep and I'm still unsure why this is happening. Here are some questions I have:

• is there even a way to install BSD grep on Linux? There seem to be a lot of people wanting to use GNU grep on BSD but I can't find any example of the opposite. (I do not have a Mac.)
• is there some way to check if /bin/grep is BSD or GNU? I guess I could check the output of grep --version but without BSD grep I can't compare it... Something like grep --version | grep GNU might work.
• is GNU grep always available on Macs or is it optional?

[polm]

@srctaha Can you upload the 8962 entries that BSD grep picks up as a gist or something so I can check it?

[srctaha]

@polm Here is the gist that lists the 8962 entries that grep (BSD grep) 2.5.1-FreeBSD picks up for grep -Ev '^([A-Za-zＡ-Ｚａ-ｚ]|[0-9０-９]*),' lex.csv.

Settings are same as before.

[polm]

Thanks, that's helpful. The results contain punctuation, kaomoji, and alphabetic entries with more than one character, which is weird... I'll see if I can figure out what's up.

[srctaha]

FYI, given line="合," or line="あい,", or line=明空,

• grep (BSD grep) 2.5.1-FreeBSD

  echo $line | grep -E '^([Ａ-Ｚ]*),' -  # match
  echo $line | grep -E '^([ａ-ｚ]*),' -  # match
  

• ggrep (GNU grep) 3.6

  echo $line | ggrep -E '^([Ａ-Ｚ]*),' -  # no match
  echo $line | ggrep -E '^([ａ-ｚ]*),' -  # no match
  

[srctaha]

is there some way to check if /bin/grep is BSD or GNU?

grep -V works for both BSD and GNU grep.

is GNU grep always available on Macs or is it optional?

For at least macOS Big Sur 11.2.1, GNU grep is unavailable by default. Users need to download it through brew, which is a free and open-source software package management system for macos.

[polm]

Thanks for the details about Macs. Since installing GNU grep isn't that hard, and since I'm still not sure exactly what's going on here, I'll just leave this alone for now, but I may come back to it later.