UnicodeEncodeError: 'ascii' codec can't encode character in __call_filer_cmd

[marxin]

@brobecke, @jakubjelinek
The issue is pretty much similar to #13 and it happens here:

'ascii' codec can't encode character u'\u0161' in position 301: ordinal not in range(128)
Traceback (most recent call last):
  File "hooks/post_receive.py", line 87, in post_receive
  File "/sourceware1/projects/src-home/git-hooks/hooks/daemon.py", line 83, in run_in_daemon
    fun()
  File "/sourceware1/projects/src-home/git-hooks/hooks/updates/emails.py", line 109, in flush
    email.send()
  File "/sourceware1/projects/src-home/git-hooks/hooks/updates/emails.py", line 355, in send
    self.__call_filer_cmd()
  File "/sourceware1/projects/src-home/git-hooks/hooks/updates/emails.py", line 456, in __call_filer_cmd
    out, _ = p.communicate(to_be_filed)
  File "/usr/lib64/python2.7/subprocess.py", line 483, in communicate
    return self._communicate(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1124, in _communicate
    stdout, stderr = self._communicate_with_poll(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1188, in _communicate_with_poll
    input_offset += os.write(fd, chunk)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u0161' in position 301: ordinal not in range(128)

we know about the input:

email_body type==<type 'unicode'>diff type==<type 'unicode'>

it happens for commits that contain e.g. š character:
https://gcc.gnu.org/pipermail/gcc-cvs/2021-January/340233.html

commit r10-5883-g6f1f3c3c5bf080a55aebdc02ed4be5ea48fbc909
Author: Martin Liska <mliska@suse.cz>
Date:   Wed Jan 6 14:14:11 2021 +0100

    Test again.
    
    Uroš
    
            PR target/98559
            * ipa-icf.c: Modify.

[brobecke]

Hello @marxin . Thanks for reporting this issue. I was able to reproduce it, and I have a tentative fix. I will let you know when the fix is available.

[marxin]

Thank you Joel for the investigation.

[brobecke]

Hello @marxin . Just to confirm that this was fixed via:

commit 1f9082567e82788e4da748fe13ddd0b230166faf
Author: Joel Brobecker <brobecker@adacore.com>
Date:   Mon Jan 11 12:53:53 2021 +0400
Subject: fix UnicodeEncodeError crash while calling file-commit-cmd hook

I updated the clone on sourceware.org, so GCC should have the fix (this assumes that GCC and GDB use the same clone of that repository -- if not, can you ask whoever maintains it to do an update?)

Thanks again for the report.

[marxin]

Thank your for the fix. I can confirm that email is sent to bugzilla, but the name is screwed somehow:

commit ab192e494fe27ee73eb3e5271f951a0e9ee02b9c (HEAD -> master, origin/master, origin/HEAD)
Author: Martin Liska <mliska@suse.cz>
Date:   Fri Jan 15 11:47:19 2021 +0100

    Add new line
    
    Uroš
    
            PR target/98559
            * ipa-icf.c: Add new line.

Email is sent here:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98559#c4

where I can see Uroš. Can you please take a look?

And yes, all the projects do share the instance of git-hooks.

[brobecke]

Can you verify it isn't the commit-filer that's causing the problem?
I suspect the filer is not decoding the data, or decoding it using the wrong encoding.

[marxin]

Well, it goes through a Python 3 script that should be fine, but it later calls a Perl script.
So for now, I would leave it..

[brobecke]

Well, it goes through a Python 3 script that should be fine, but it later calls a Perl script.

My knowledge and understanding of string-vs-bytes as well as encodings, etc, is still fairly raw, but I think that the fact that the receiving script is written in Python 3 unfortunately doesn't really guarantee that the data is read correctly by your (Python 3) script. What's happening is that the git-hooks is sending the data in byte form down to the filer script, and the filer has to read that byte-stream and decode it using the proper encoding. A lot of times, as was actually the case in the crash reported in this issue, unless done explicitly the conversion actually happens implicitly as need, in which case a default encoding gets used. Luckily with Python 3.x, I believe the default encoding is UTF-8 (whereas it's ASCII with Python 2.7). So despite my warning here, I have good hopes that the filer script should be decoding the data correctly thanks to it being Python3 and the default encoding being UTF-8.

The same scenario happens again at the interface between the filer and the perl script. Normally, the spirit in Python 3 is to do all it can to force you to encode your data back in byte form before you want to output a string, and thus before you send it to the perl script. When doing so, you either do the conversion explicitly with the encoding of your choice, or else rely on Python's default. From there, your perl script will have to do the reverse conversion which consists of decoding the byte string with the same encoding as the one used by your Python script.

So for now, I would leave it..

OK. I think what I will do, though, is write myself an Action Item to think about formalizing a bit more the encoding being used by the hooks at its interface with the various project-specific config hooks, so that there is less uncertainty around what to use... For when I have another weekend with a bit of free time...