New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unknown encoding: 65001 #2009

Open
PetaPetaPeta opened this Issue Apr 26, 2017 · 24 comments

Comments

Projects
None yet
@PetaPetaPeta

PetaPetaPeta commented Apr 26, 2017

I am attempting to run docker-compose.exe from Windows in BashOnWindows. Specifically I am running docker-compose.exe --version to verify that it can correctly output data. When doing this I am receiving the following error:

~ # docker-compose.exe --version                                                                                                                                                                                                  
Traceback (most recent call last):
  File "docker-compose", line 3, in <module>
  File "compose\cli\main.py", line 66, in main
  File "compose\cli\main.py", line 98, in dispatch
  File "compose\cli\docopt_command.py", line 25, in parse
  File "compose\cli\docopt_command.py", line 12, in docopt_full_help
  File "site-packages\docopt.py", line 575, in docopt
  File "site-packages\docopt.py", line 484, in extras
LookupError: unknown encoding: cp65001
Failed to execute script docker-compose
------------------------------------------------------------

I can see that codepage 65001 is the Windows codepage for UTF-8 encoding, but that this is not supported in Unix systems. A solution to this is to run the command chcp.com 437 to change to a different codepage. This solution is however not good as it removes the support for UTF-8 in the terminal.
The issue should be persistent across different Windows applications that attempt to output any text in encoding 65001.

Windows build number: Microsoft Windows [Version 10.0.15063]

@sgtoj

This comment has been minimized.

Show comment
Hide comment
@sgtoj

sgtoj Apr 27, 2017

Now that we understand the problem is encoding related. I am not sure if this an issue with the Windows Console, WSL, Python, or Docker Compose (which is compiled Py 2/3 code)?

sgtoj commented Apr 27, 2017

Now that we understand the problem is encoding related. I am not sure if this an issue with the Windows Console, WSL, Python, or Docker Compose (which is compiled Py 2/3 code)?

@CherryDT

This comment has been minimized.

Show comment
Hide comment
@CherryDT

CherryDT Apr 27, 2017

This issue also causes display issues with Windows programs run from within bash.

See missing/incorrect umlauts below:

image

My codepage is 850, but the inner cmd.exe's (ran from bash) is 65001.

Running chcp 850 in the inner cmd fixes it.

What is strange though is the following, why is it like that?

  • Open cmd
    • Run chcp - output: 850
    • Run bash
      • Run cmd.exe /c chcp - output: 65001 (expected because of this issue, albeit not good)
      • Run cmd.exe
        • Run chcp - output: 65001 (expected)
        • Run chcp 437
        • Run chcp - output: 437 (expected)
        • Run exit (back to bash)
      • Run cmd.exe /c chcp - output: 437 (why did the chcp inside the Windows child process have an effect on future child processes spawned by its Linux parent?)
      • Run exit (back to outer cmd)
    • Run chcp - output: 850 (looking at the previous 437 output, why is it not changed here as well?)

CherryDT commented Apr 27, 2017

This issue also causes display issues with Windows programs run from within bash.

See missing/incorrect umlauts below:

image

My codepage is 850, but the inner cmd.exe's (ran from bash) is 65001.

Running chcp 850 in the inner cmd fixes it.

What is strange though is the following, why is it like that?

  • Open cmd
    • Run chcp - output: 850
    • Run bash
      • Run cmd.exe /c chcp - output: 65001 (expected because of this issue, albeit not good)
      • Run cmd.exe
        • Run chcp - output: 65001 (expected)
        • Run chcp 437
        • Run chcp - output: 437 (expected)
        • Run exit (back to bash)
      • Run cmd.exe /c chcp - output: 437 (why did the chcp inside the Windows child process have an effect on future child processes spawned by its Linux parent?)
      • Run exit (back to outer cmd)
    • Run chcp - output: 850 (looking at the previous 437 output, why is it not changed here as well?)
@sgtoj

This comment has been minimized.

Show comment
Hide comment
@sgtoj

sgtoj Apr 27, 2017

Oddly, it says cmd.exe doesn't exists on Windows 10.0.16179.

sgtoj commented Apr 27, 2017

Oddly, it says cmd.exe doesn't exists on Windows 10.0.16179.

@sunjoong

This comment has been minimized.

Show comment
Hide comment
@sunjoong

sunjoong Apr 27, 2017

@sgtoj - cmd.exe shoud exist, but stragne if not exist!! Check with ls command; "ls /mnt/c/Windows/System32/cmd.exe".

sunjoong commented Apr 27, 2017

@sgtoj - cmd.exe shoud exist, but stragne if not exist!! Check with ls command; "ls /mnt/c/Windows/System32/cmd.exe".

@PetaPetaPeta

This comment has been minimized.

Show comment
Hide comment
@PetaPetaPeta

PetaPetaPeta May 2, 2017

@sgtoj From my investigations the issue is that the Python installed in WSL does not support the code page 65001. You can see this if you dig around a bit in /usr/lib/python2.7/encodings/.

I am using cmder and the program can be run in the same terminal if I set it to use powershell. It is thus not isolated to docker-compose and I suspect it will be like this for any program that is compiled using codepage 65001.
That leaves it to whether it is a thing that should be supported in WSL. My opinion on this matter is that it should as one of the features of WSL is running Windows programs in bash.

PetaPetaPeta commented May 2, 2017

@sgtoj From my investigations the issue is that the Python installed in WSL does not support the code page 65001. You can see this if you dig around a bit in /usr/lib/python2.7/encodings/.

I am using cmder and the program can be run in the same terminal if I set it to use powershell. It is thus not isolated to docker-compose and I suspect it will be like this for any program that is compiled using codepage 65001.
That leaves it to whether it is a thing that should be supported in WSL. My opinion on this matter is that it should as one of the features of WSL is running Windows programs in bash.

@MZAWeb

This comment has been minimized.

Show comment
Hide comment
@MZAWeb

MZAWeb Jun 9, 2017

+1 here with the same issue

MZAWeb commented Jun 9, 2017

+1 here with the same issue

@sunjoong

This comment has been minimized.

Show comment
Hide comment
@sunjoong

sunjoong Jun 10, 2017

chcp

From my investigations the issue is that the Python installed in WSL does not support the code page 65001.

@PetaPetaPeta - No. Python within docker-compose does not support code page 65001 makes this problem, not python in WSL.
@MZAWeb - Command after cmd.exe /C chcp 437 or other code page.

sunjoong commented Jun 10, 2017

chcp

From my investigations the issue is that the Python installed in WSL does not support the code page 65001.

@PetaPetaPeta - No. Python within docker-compose does not support code page 65001 makes this problem, not python in WSL.
@MZAWeb - Command after cmd.exe /C chcp 437 or other code page.

@BennettStaley

This comment has been minimized.

Show comment
Hide comment
@BennettStaley

BennettStaley Jun 26, 2017

+1 here with the same issue. Cant compose anything.

BennettStaley commented Jun 26, 2017

+1 here with the same issue. Cant compose anything.

@sunjoong

This comment has been minimized.

Show comment
Hide comment
@sunjoong

sunjoong Jun 26, 2017

I had shown you can run docker-compose.exe --version on WSL on the previous comment. I'm not a docker user, so I did not go further, but I'm curious that works for you and why you do not use https://github.com/docker/compose/releases/download/1.14.0/docker-compose-Linux-x86_64 .

sunjoong commented Jun 26, 2017

I had shown you can run docker-compose.exe --version on WSL on the previous comment. I'm not a docker user, so I did not go further, but I'm curious that works for you and why you do not use https://github.com/docker/compose/releases/download/1.14.0/docker-compose-Linux-x86_64 .

@whitecolor

This comment has been minimized.

Show comment
Hide comment
@whitecolor

whitecolor Jun 29, 2017

I run such thing using custom linux executables that contain code like this:
cmd.exe /c "chcp 850 >> nul && docker-compose $@"

The same thing goed for multipe python based apps

whitecolor commented Jun 29, 2017

I run such thing using custom linux executables that contain code like this:
cmd.exe /c "chcp 850 >> nul && docker-compose $@"

The same thing goed for multipe python based apps

@whitecolor

This comment has been minimized.

Show comment
Hide comment
@whitecolor

whitecolor Jun 29, 2017

Yes, was mistype, thanks.

whitecolor commented Jun 29, 2017

Yes, was mistype, thanks.

@lfurzewaddock

This comment has been minimized.

Show comment
Hide comment
@lfurzewaddock

lfurzewaddock Jul 17, 2017

OS: MS Windows 10 Pro - Version: 10.0.15063 Build 15063 - Bash on Ubuntu on Windows (WSL)
Docker CE for Windows - Version 17.06.0-ce-win19 (12801) - Channel: stable c98c1c2
ConEmu console emulator: 161206 [64] {Stable}

To determine what 'active code page' the standard MS Windows CMD console is using, open an MS Windows CMD prompt, then type:

chcp

My machine (British English) returned the code for Multilingual (Latin I):

Active code page: 850

After discovering the current 'active code page' is 850 in the ConEmu console emulator type:

cmd.exe /c chcp

This verified the current 'active code page' the ConEmu console emulator is using to be:

Active code page: 65001

To set the 'active code page' to be 850 in the ConEmu console emulator type;

cmd.exe /c chcp 850

Which responds with;

Active code page: 850

Finally the command...

docker-compose -version

...responds as expected with;

docker-compose version 1.14.0, build c7bdf9e3

Although, I haven't yet worked out how to make this permanent as I expect the next time I restart I will have to repeat the previous steps?

@sunjoong correct the change was not permanent.

Permanently apply this setting to all future bash sessions by adding the following to your ~/.bashrc file

# Set Multilingual (Latin I) aka Code Page 850 (Workaround issue with docker-compose and utf8 incompatibility)

chcp.com 850

It seems this causes other problems such as the output of the npm install dependency tree, so it appears the safest way is to change the code page to 850, before running docker-compose and then returning the code page back to the original value.

Does anyone know how to automate this?

OK, inspired by this GitHub post: Unicode Display Error (discovered via npm STDOUT) #3819 a workaround is to pipe the output to a string, eg;

docker-compose version | while read line; do echo $line; done

will output...

docker-compose version 1.14.0, build c7bdf9e3
docker-py version: 2.3.0
CPython version: 2.7.13
OpenSSL version: OpenSSL 1.0.2j 26 Sep 2016

This is a known bug: GitHub post: Unknown encoding error in Windows with cp65001 #2775.

Hopefully it will be fixed soon.

lfurzewaddock commented Jul 17, 2017

OS: MS Windows 10 Pro - Version: 10.0.15063 Build 15063 - Bash on Ubuntu on Windows (WSL)
Docker CE for Windows - Version 17.06.0-ce-win19 (12801) - Channel: stable c98c1c2
ConEmu console emulator: 161206 [64] {Stable}

To determine what 'active code page' the standard MS Windows CMD console is using, open an MS Windows CMD prompt, then type:

chcp

My machine (British English) returned the code for Multilingual (Latin I):

Active code page: 850

After discovering the current 'active code page' is 850 in the ConEmu console emulator type:

cmd.exe /c chcp

This verified the current 'active code page' the ConEmu console emulator is using to be:

Active code page: 65001

To set the 'active code page' to be 850 in the ConEmu console emulator type;

cmd.exe /c chcp 850

Which responds with;

Active code page: 850

Finally the command...

docker-compose -version

...responds as expected with;

docker-compose version 1.14.0, build c7bdf9e3

Although, I haven't yet worked out how to make this permanent as I expect the next time I restart I will have to repeat the previous steps?

@sunjoong correct the change was not permanent.

Permanently apply this setting to all future bash sessions by adding the following to your ~/.bashrc file

# Set Multilingual (Latin I) aka Code Page 850 (Workaround issue with docker-compose and utf8 incompatibility)

chcp.com 850

It seems this causes other problems such as the output of the npm install dependency tree, so it appears the safest way is to change the code page to 850, before running docker-compose and then returning the code page back to the original value.

Does anyone know how to automate this?

OK, inspired by this GitHub post: Unicode Display Error (discovered via npm STDOUT) #3819 a workaround is to pipe the output to a string, eg;

docker-compose version | while read line; do echo $line; done

will output...

docker-compose version 1.14.0, build c7bdf9e3
docker-py version: 2.3.0
CPython version: 2.7.13
OpenSSL version: OpenSSL 1.0.2j 26 Sep 2016

This is a known bug: GitHub post: Unknown encoding error in Windows with cp65001 #2775.

Hopefully it will be fixed soon.

@sunjoong

This comment has been minimized.

Show comment
Hide comment
@sunjoong

sunjoong Jul 18, 2017

I will have to repeat the previous steps?

@PetaPetaPetalfurzewaddock - I think so.

sunjoong commented Jul 18, 2017

I will have to repeat the previous steps?

@PetaPetaPetalfurzewaddock - I think so.

@veqryn

This comment has been minimized.

Show comment
Hide comment
@veqryn

veqryn Jul 28, 2017

I've added cmd.exe /c chcp 437 > /tmp/chcp 2>&1 to my .bashrc file in WSL as a workaround until this is fixed. (437 being what cmd.exe prints out when it runs chcp on my laptop)

veqryn commented Jul 28, 2017

I've added cmd.exe /c chcp 437 > /tmp/chcp 2>&1 to my .bashrc file in WSL as a workaround until this is fixed. (437 being what cmd.exe prints out when it runs chcp on my laptop)

@veqryn

This comment has been minimized.

Show comment
Hide comment
@veqryn

veqryn Jul 29, 2017

Nevermind, it seems that workaround has its own issues: It messes up characters.

11:58:02 ~$ cmd.exe /c chcp 437 > /tmp/chcp 2>&1
11:58:06 ~$ sudo find / -iname "pkg.?config"
find: ΓÇÿ/mnt/c/$Recycle.Bin/S-1-5-18ΓÇÖ: Permission denied

11:58:12 ~$ cmd.exe /c chcp 65001 > /tmp/chcp 2>&1
11:58:19 ~$ sudo find / -iname "pkg.?config"
find: ‘/mnt/c/$Recycle.Bin/S-1-5-18’: Permission denied

11:58:23 ~$ docker-compose stop
Traceback (most recent call last):
  File "logging\__init__.py", line 872, in emit
LookupError: unknown encoding: cp65001
Logged from file main.py, line 74

veqryn commented Jul 29, 2017

Nevermind, it seems that workaround has its own issues: It messes up characters.

11:58:02 ~$ cmd.exe /c chcp 437 > /tmp/chcp 2>&1
11:58:06 ~$ sudo find / -iname "pkg.?config"
find: ΓÇÿ/mnt/c/$Recycle.Bin/S-1-5-18ΓÇÖ: Permission denied

11:58:12 ~$ cmd.exe /c chcp 65001 > /tmp/chcp 2>&1
11:58:19 ~$ sudo find / -iname "pkg.?config"
find: ‘/mnt/c/$Recycle.Bin/S-1-5-18’: Permission denied

11:58:23 ~$ docker-compose stop
Traceback (most recent call last):
  File "logging\__init__.py", line 872, in emit
LookupError: unknown encoding: cp65001
Logged from file main.py, line 74
@veqryn

This comment has been minimized.

Show comment
Hide comment
@veqryn

veqryn Jul 29, 2017

Anyone know how to hook the linux version of docker-compose up to the windows version of docker?
Even if I put them in the same directory (in the mnt) I get this:

12:04:02 ~$ docker-compose logs
ERROR: Couldn't connect to Docker daemon. You might need to install Docker:

https://docs.docker.com/engine/installation/

veqryn commented Jul 29, 2017

Anyone know how to hook the linux version of docker-compose up to the windows version of docker?
Even if I put them in the same directory (in the mnt) I get this:

12:04:02 ~$ docker-compose logs
ERROR: Couldn't connect to Docker daemon. You might need to install Docker:

https://docs.docker.com/engine/installation/
@veqryn

This comment has been minimized.

Show comment
Hide comment
@veqryn

veqryn Jul 29, 2017

Nevermind, its bash so I just made a function to pipe the output to cat.
Turns out as long as stdout and stderr go to something else before they go to the console, python doesn't complain.

docker-compose() {
  command docker-compose.exe "$@" 2>&1 | cat
}

veqryn commented Jul 29, 2017

Nevermind, its bash so I just made a function to pipe the output to cat.
Turns out as long as stdout and stderr go to something else before they go to the console, python doesn't complain.

docker-compose() {
  command docker-compose.exe "$@" 2>&1 | cat
}
@whitecolor

This comment has been minimized.

Show comment
Hide comment
@whitecolor

whitecolor Jul 29, 2017

I would really recommend using Linux native clients for docker and docker-compose, they are easy to install and it will keep you from many problems with executing windows natives in WSL.

whitecolor commented Jul 29, 2017

I would really recommend using Linux native clients for docker and docker-compose, they are easy to install and it will keep you from many problems with executing windows natives in WSL.

@veqryn

This comment has been minimized.

Show comment
Hide comment
@veqryn

veqryn Jul 29, 2017

@whitecolor besides issues with the console (ie: #2379), what issues are there with running windows executables from inside WSL?
My plan is to do everything from inside wsl/bash...

veqryn commented Jul 29, 2017

@whitecolor besides issues with the console (ie: #2379), what issues are there with running windows executables from inside WSL?
My plan is to do everything from inside wsl/bash...

@MatthewMiele

This comment has been minimized.

Show comment
Hide comment
@MatthewMiele

MatthewMiele Jul 29, 2017

@whitecolor the only issue with running native Linux version on WSL is the issue with paths. This is the original issue I raised before I realised it isn't actually a 'bug'. docker/for-win#298

@veqryn I need to try again but I seem to remember having performance issues when trying to use docker-compose.exe from WSL. It took several seconds for the commands to respond.

MatthewMiele commented Jul 29, 2017

@whitecolor the only issue with running native Linux version on WSL is the issue with paths. This is the original issue I raised before I realised it isn't actually a 'bug'. docker/for-win#298

@veqryn I need to try again but I seem to remember having performance issues when trying to use docker-compose.exe from WSL. It took several seconds for the commands to respond.

@veqryn

This comment has been minimized.

Show comment
Hide comment
@veqryn

veqryn Jul 29, 2017

I remount the /mnt/c to /c and that solves the issues with docker & compose mounting volumes.

veqryn commented Jul 29, 2017

I remount the /mnt/c to /c and that solves the issues with docker & compose mounting volumes.

@mcuyar

This comment has been minimized.

Show comment
Hide comment
@mcuyar

mcuyar Sep 6, 2017

For anyone who might run into this, I was able to find a work around using WSL. I use zsh, and added this to my .zshrc, but it should work equally as well in .bashrc

function docker_compose() {
    echo "$(docker-compose.exe ${@})"
}

alias docker-compose="docker_compose ${@}"

mcuyar commented Sep 6, 2017

For anyone who might run into this, I was able to find a work around using WSL. I use zsh, and added this to my .zshrc, but it should work equally as well in .bashrc

function docker_compose() {
    echo "$(docker-compose.exe ${@})"
}

alias docker-compose="docker_compose ${@}"
@gempir

This comment has been minimized.

Show comment
Hide comment
@gempir

gempir Sep 12, 2017

My workaround for this: Create a file at /usr/bin/docker-compose
and put this in:

#!/usr/bin/env bash

trap "/mnt/c/Windows/System32/cmd.exe /C chcp 65001" INT

/mnt/c/Windows/System32/cmd.exe /C chcp 850

/mnt/c/Program\ Files/Docker/Docker/resources/bin/docker-compose.exe "$@"

/mnt/c/Windows/System32/cmd.exe /C chcp 65001

now I can run docker-compose from Bash on Ubuntu on Windows without any encoding issues.

gempir commented Sep 12, 2017

My workaround for this: Create a file at /usr/bin/docker-compose
and put this in:

#!/usr/bin/env bash

trap "/mnt/c/Windows/System32/cmd.exe /C chcp 65001" INT

/mnt/c/Windows/System32/cmd.exe /C chcp 850

/mnt/c/Program\ Files/Docker/Docker/resources/bin/docker-compose.exe "$@"

/mnt/c/Windows/System32/cmd.exe /C chcp 65001

now I can run docker-compose from Bash on Ubuntu on Windows without any encoding issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment