fish_update_completions throws UnicodeDecodeError

[nmeum]

manpath: warning: $MANPATH set, inserting /etc/manpath.config
Parsing man pages and writing completions to /home/nmeum/.config/fish/generated_completions/
Traceback (most recent call last):                 
  File "/usr/local/share/fish/tools/create_manpage_completions.py", line 878, in parse_and_output_man_pages
    if parse_manpage_at_path(manpage_path, output_directory):
  File "/usr/local/share/fish/tools/create_manpage_completions.py", line 794, in parse_manpage_at_path
    success = parser.parseManPage(manpage)
  File "/usr/local/share/fish/tools/create_manpage_completions.py", line 676, in parseManPage
    built_command(options, description)
  File "/usr/local/share/fish/tools/create_manpage_completions.py", line 170, in built_command
    output_complete_command(escaped_cmd, fish_options, truncated_description, built_command_output)
  File "/usr/local/share/fish/tools/create_manpage_completions.py", line 102, in output_complete_command
    output_list.append(' '.join(comps))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 7: ordinal not in range(128)
  1185 / 1185 : zsoelim.1.gz

[siteshwar]

How old is the build of fish which you're running ? I believe this issue was fixed by #420

[nmeum]

It is pretty new - the last commit is bcab703

[ridiculousfish]

Can you please post somewhere the verbose output, that is, run this:

/usr/local/share/fish/tools/create_manpage_completions.py --manpath --verbose 2

Thanks!

[nmeum]

@ridiculousfish https://gist.github.com/nmeum/5382366

[ridiculousfish]

Perfect, thanks.

Looks like nping.1.gz tripped us up.

[ridiculousfish]

Should be fixed here: 1c55563

Thanks for reporting this

[dag]

I'm still seeing this with master HEAD.

Considering /usr/share/man/man1/btparser.1.gz
   Trying parser Type2
      Command is btparser
      Error parsing /usr/share/man/man1/btparser.1.gz: <type 'exceptions.UnicodeDecodeError'>
Traceback (most recent call last):
  File "/usr/local/share/fish/tools/create_manpage_completions.py", line 894, in parse_and_output_man_pages
    if parse_manpage_at_path(manpage_path, output_directory):
  File "/usr/local/share/fish/tools/create_manpage_completions.py", line 810, in parse_manpage_at_path
    success = parser.parseManPage(manpage)
  File "/usr/local/share/fish/tools/create_manpage_completions.py", line 414, in parseManPage
    built_command(optionName, optionDescription)
  File "/usr/local/share/fish/tools/create_manpage_completions.py", line 180, in built_command
    truncated_description = truncated_description[:prefix_len] + truncation_suffix
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 13: ordinal not in range(128)

[nmeum]

@dag Maybe you should also paste the verbose output somewhere?

/usr/local/share/fish/tools/create_manpage_completions.py --manpath --verbose 2

[dag]

@nmeum That was from the verbose output, but skipping all the irrelevant bits.

[nmeum]

oops, indeed

[Susurrus]

This happened to me as well. Same issue as dag commented like:

Considering /usr/share/man/man1/xfce4-settings-manager.1.gz
   Trying parser Type3
      Command is xfce4-settings-manager
   Trying parser Darwin man parser
   Trying parser Deroffing man parser
      Error parsing /usr/share/man/man1/xfce4-settings-manager.1.gz: <type 'exceptions.UnicodeDecodeError'>
Traceback (most recent call last):
  File "/usr/local/share/fish/tools/create_manpage_completions.py", line 894, in parse_and_output_man_pages
    if parse_manpage_at_path(manpage_path, output_directory):
  File "/usr/local/share/fish/tools/create_manpage_completions.py", line 810, in parse_manpage_at_path
    success = parser.parseManPage(manpage)
  File "/usr/local/share/fish/tools/create_manpage_completions.py", line 692, in parseManPage
    built_command(options, description)
  File "/usr/local/share/fish/tools/create_manpage_completions.py", line 180, in built_command
    truncated_description = truncated_description[:prefix_len] + truncation_suffix
UnicodeDecodeError: 'ascii' codec can't decode byte 0xa9 in position 37: ordinal not in range(128)

but with xfce4-appearance-settings, xfce4-display-settings, xfce4-keyboard-settings, xfce4-mouse-settings, xfce4-notifyd-config, xfce4-session-settings, xfce4-settings-editor, xfce-settings-help, xfce4-settings-manager, etc. The error in all of these packages is byte 0xa9 at position 37.

I'm running what was Ubuntu 12.04, but with xubuntu-desktop package installed. I'd be happy to test any potential patches on my system.

[ridiculousfish]

Hopefully fixed again by 2f50162

[dag]

Works for me!

[ridiculousfish]

Great, thanks for the quick check!

[Susurrus]

Worked for me too, thanks. As for this commit, it's on the Integration_2.0.0 branch. Is that the latest development branch and I should follow that to stay on top of things?

[ridiculousfish]

It is for the next few days, until we release, at which point it will be merged into master.

[jvoisin]

Considering /usr/share/man/man1/pdf-parser.1.gz
   Trying parser Darwin man parser
      Error parsing /usr/share/man/man1/pdf-parser.1.gz: <type 'exceptions.UnicodeDecodeError'>
Traceback (most recent call last):
  File "/usr/share/fish/tools/create_manpage_completions.py", line 897, in parse_and_output_man_pages
    if parse_manpage_at_path(manpage_path, output_directory):
  File "/usr/share/fish/tools/create_manpage_completions.py", line 813, in parse_manpage_at_path
    success = parser.parseManPage(manpage)
  File "/usr/share/fish/tools/create_manpage_completions.py", line 627, in parseManPage
    desc = ' '.join(desc_lines)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 41: ordinal not in range(128)

and

Considering /usr/share/man/man1/pdfid.1.gz
   Trying parser Darwin man parser
      Error parsing /usr/share/man/man1/pdfid.1.gz: <type 'exceptions.UnicodeDecodeError'>
Traceback (most recent call last):
  File "/usr/share/fish/tools/create_manpage_completions.py", line 897, in parse_and_output_man_pages
    if parse_manpage_at_path(manpage_path, output_directory):
  File "/usr/share/fish/tools/create_manpage_completions.py", line 813, in parse_manpage_at_path
    success = parser.parseManPage(manpage)
  File "/usr/share/fish/tools/create_manpage_completions.py", line 627, in parseManPage
    desc = ' '.join(desc_lines)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 16: ordinal not in range(128)

On fish 2.0.0.