Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug]: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 59: invalid continuation byte #6105

Closed
1 task done
cmensah opened this issue Apr 2, 2024 · 8 comments · Fixed by #6133
Closed
1 task done
Labels
bug Something isn't working

Comments

@cmensah
Copy link

cmensah commented Apr 2, 2024

Is there an existing issue for this problem?

  • I have searched the existing issues

Operating system

Windows

GPU vendor

Nvidia (CUDA)

GPU model

RTX 3050

GPU VRAM

8GB

Version number

4.0.0

Browser

Chrome

Python dependencies

No response

What happened

I was trying to update to the latest version.. then this error :
***** Checking and Updating Python *****
Initializing the installer. This may take a minute - please wait...
Actual environment location may have moved due to redirects, links or junctions.
Requested location: "C:\Users\SOMBRE~1\AppData\Local\Temp\invokeai-installer-tmpwb2fytyg\Scripts\python.exe"
Actual location: "C:\Users\Sombre Clarté\AppData\Local\Temp\invokeai-installer-tmpwb2fytyg\Scripts\python.exe"
Traceback (most recent call last):
File "C:\Users\Sombre Clarté\Downloads\InvokeAI-installer-v4.0.0\InvokeAI-Installer\lib\main.py", line 50, in
inst = Installer()
^^^^^^^^^^^
File "C:\Users\Sombre Clarté\Downloads\InvokeAI-installer-v4.0.0\InvokeAI-Installer\lib\installer.py", line 48, in init
self.bootstrap()
File "C:\Users\Sombre Clarté\Downloads\InvokeAI-installer-v4.0.0\InvokeAI-Installer\lib\installer.py", line 88, in bootstrap
res = upgrade_pip(Path(venv_dir.name))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Sombre Clarté\Downloads\InvokeAI-installer-v4.0.0\InvokeAI-Installer\lib\installer.py", line 319, in upgrade_pip
result = subprocess.check_output([python, "-m", "pip", "install", "--upgrade", "pip"]).decode()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 59: invalid continuation byte
Appuyez sur une touche pour continuer...

What you expected to happen

Version 4.0.0

How to reproduce the problem

Start the install.bat

Additional context

No response

Discord username

No response

@cmensah cmensah added the bug Something isn't working label Apr 2, 2024
@hipsterusername
Copy link
Member

@lstein - Do you think this might be system specific?

@cmensah
Copy link
Author

cmensah commented Apr 2, 2024

Maybe i need to add this :
-my system is in French

  • my python version is 3.11.5
  • my pip version is 24.0
    Previous InvokeAI version was working great

@hipsterusername
Copy link
Member

Maybe i need to add this : -my system is in French

* my python version is 3.11.5

* my pip version is 24.0
  Previous InvokeAI version was working great

Yes, that seems to be the case - But that's an issue we need to fix I think

@psychedelicious
Copy link
Collaborator

It appears the é character in Sombre Clarté is the issue.

Please open a command line window and try this command. Does it error?

python -c "import subprocess; print(subprocess.check_output('echo Sombre Clarté', shell=True).decode())"

Next, try this command. Does it error?

python -c "import subprocess; print(subprocess.check_output('echo Sombre Clarté', shell=True).decode("utf-8"))"

@cmensah
Copy link
Author

cmensah commented Apr 3, 2024

Okay,.
First :
C:\Users\Sombre Clarté>python -c "import subprocess; print(subprocess.check_output('echo Sombre Clarté', shell=True).decode())"
Traceback (most recent call last):
File "", line 1, in
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 12: invalid start byte

C:\Users\Sombre Clarté>python -c "import subprocess; print(subprocess.check_output('echo Sombre Clarté', shell=True).decode())"
Traceback (most recent call last):
File "", line 1, in
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 12: invalid start byte

Second :

C:\Users\Sombre Clarté>python -c "import subprocess; print(subprocess.check_output('echo Sombre Clarté', shell=True).decode("utf-8"))"
Traceback (most recent call last):
File "", line 1, in
NameError: name 'utf' is not defined


Indeed, the character "é" has caused me some problems since I started to get interested in scripts and everything related to artificial intelligence.

I have been using my windows system for far too long, so it's too late to change.
I'll think about it if I ever install another system!

@psychedelicious
Copy link
Collaborator

@cmensah I apologize, I had a typo in the second command - please try this one:

python -c "import subprocess; print(subprocess.check_output('echo Sombre Clarté', shell=True).decode('utf-8'))"

Hopefully this works without an error, and we can update the installer to handle é.

@cmensah
Copy link
Author

cmensah commented Apr 3, 2024

@cmensah I apologize, I had a typo in the second command - please try this one:

python -c "import subprocess; print(subprocess.check_output('echo Sombre Clarté', shell=True).decode('utf-8'))"

Hopefully this works without an error, and we can update the installer to handle é.


Microsoft Windows [version 10.0.19045.3693]
(c) Microsoft Corporation. Tous droits réservés.

C:\Users\Sombre Clarté>python -c "import subprocess; print(subprocess.check_output('echo Sombre Clarté', shell=True).decode('utf-8'))"
Traceback (most recent call last):
File "", line 1, in
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 12: invalid start byte

C:\Users\Sombre Clarté>

this guy : 0x82 :)

@psychedelicious
Copy link
Collaborator

Oh boy, fun.

We have another related issue with an unexpected encoding here: https://discord.com/channels/1020123559063990373/1020123559831539744/1225085570032865291

In this case, the user (also with French characters) is getting an error when we attempt to read their config file into memory:

image

psychedelicious added a commit that referenced this issue Apr 4, 2024
We have had a few bugs with v4 related to file encodings, especially on Windows.

Windows uses its own character encodings instead of `utf-8`, often `cp1252`. Some characters cannot be decoded using `utf-8`, causing `UnicodeDecodeError`.

There are a couple places where this can cause problems:
- In the installer bootstrap, we install or upgrade `pip` and decode the result, using `subprocess`.

  The input to this includes the user's home dir. In #6105, the user had one of the problematic characters in their username. `subprocess` attempts and fails to decode the username, which crashes the installer.

  To fix this, we need to use `locale.getpreferredencoding()` when executing the command.
- Similarly, in the model install service and config class, we attempt to load a yaml config file. If a problematic character is in the path to the file (which often includes the user's home dir), we can get the same error.

  One example is  #6129 in which the models.yaml migration fails.

  To fix this, we need to open the file with `locale.getpreferredencoding()`.
psychedelicious added a commit that referenced this issue Apr 4, 2024
We have had a few bugs with v4 related to file encodings, especially on Windows.

Windows uses its own character encodings instead of `utf-8`, often `cp1252`. Some characters cannot be decoded using `utf-8`, causing `UnicodeDecodeError`.

There are a couple places where this can cause problems:
- In the installer bootstrap, we install or upgrade `pip` and decode the result, using `subprocess`.

  The input to this includes the user's home dir. In #6105, the user had one of the problematic characters in their username. `subprocess` attempts and fails to decode the username, which crashes the installer.

  To fix this, we need to use `locale.getpreferredencoding()` when executing the command.
- Similarly, in the model install service and config class, we attempt to load a yaml config file. If a problematic character is in the path to the file (which often includes the user's home dir), we can get the same error.

  One example is  #6129 in which the models.yaml migration fails.

  To fix this, we need to open the file with `locale.getpreferredencoding()`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants