Colorizing doesn't work on Windows with git-bash

[julie777]

Description of your problem

Please provide a minimal, self-contained, and reproducible example.

putup --veryverbose tmpproj

Please provide the full traceback using the --very-verbose flag.
Output has no color.

Please provide any additional information below.
istty check is failing, probably because of the know bug in pip
we are also checking for curses and colorama
I understand the check for curses as it is part of the standard library.
However, if we want to use colorama we should be installing it.

Versions and main components

• PyScaffold Version: 4.1.4
• Python Version: 3.10.2
• Operating system: Windows/git-bash
• How did you install PyScaffold: pipx

[julie777]

I just verified that istty is succeeding and I still have no color

[abravalheri]

Hi @julie777, I think the best way of fixing this is to add a conditional dependency on colorama for Windows.

At least that is what this line indicates:

pyscaffold/src/pyscaffold/termui.py

Line 80 in c4469f0

SYSTEM_SUPPORTS_COLOR = curses_available() or init_colorama()

Can you check what is the value of this variable, and also what is the effect of installing colorama in the same virtualenv you have PyScaffold installed?

(If in the future we move towards rich probably this would be already handled)

[julie777]

I will certainly investigate more.

(One of the reasons that I am working in the git-bash on Windows environment, instead of just developing on Linux which would make everything easy, is to see how many edge cases that I can find and fix.)

[julie777]

So it turns out there are a couple problems.

1. pip22 has a bug and when it installs packages with executables they don't get stdout and stderr connected correctly. Reverting back to pip21 fixed that problem. This caused some programs to run, and fail, with no output except the return code. I first had to debug the problem with pre-commit, then later with other tools installed into my venv.

2. By default git-bash and msys and mingw variants use mintty as the terminal for bash. The way mintty connects bash under Windows which allows running windows commands (such as python.exe) is by actually using a cmd.exe that is hidden. This causes mintty to not report being a tty. This is why --help doesn't have any word wrapping or color.
   a. There is another way to run git-bash from a non-default location which lets programs see isatty==True, but I believe it is running using the terminal that cmd.exe runs in. The results were terrible.

Number 1 will solve itself. I just documented number 2. I might try some other options later, but for now you can close the issue if you like.

[FlorianWilhelm]

@julie777 does this mean that most people under Windows will have no color? Do you see color if you are using other Python tools? Might also be that we need to change things. Or is it only a problem in git-bash, and this does something strange?

[julie777]

The problem I have been describing happens with running git-bash on window which creates a terminal using mintty. That is the way I have been running it because installation puts an entry into the start menu to run it. And, the other reason I have been using it is that git-bash and mintty work well with all my terminal (xterm) settings and customization.

I will will run a complete set of tests today

I was putting off doing a full test for every environment because I don't have everything setup perfectly to jump between them all. [Last night I worked to get more environments working]

It is surprising, but even though git-bash, msys2 and mingw64 are all from the msys codebase they have differences:

• where the Linux filesystem is mounted
• how they represent the windows drive letter
• which environment variables they set
• how they translate the windows path, adding and removing directories.

Cygwin even more differences

I had previously been working to identify each one to be able to find their differences and had collected a full set of environment variables defined in bash when running without and dot files.

Testing is harder than it should be for some of them as they don't have bash installed in the "Linux" part and can't seem to run it from the Windows install.

I have done some communicating with those communities and found ways to identify all the environments which wasn't obvious because printenv doesn't show all the variables that are available. For example the standard variable OSTYPE is defined in the cygwin shell, but not exported to the environment.

Environments that can run Windows Python, with apps and tools

• cmd.exe
• powershell
• git bash
• MSYS bash
• mingw64 bash
• cygwin bash
• Ubuntu bash WSL
  (I am skipping Gow bash because it doesn't support terminals well. No tput, etc. My dotfiles confuse it and mostly Gow is just to support running standard commands (ls, grep, find, ...) From a windows shell.

Terminals that those "shells" can run in

• Windows Console
• Windows Terminal
• mintty
• whatever the default term is that is used when opening a WSL shell
• mobaxterm

Not all shells run in all terminals, but it is still a large cross product of shell/env, terminal, color support, column wrapping.

[julie777]

(Aside: I did try git-bash in Windows terminal originally, but I am so used to having every instance open in a new window that I let the tabbed interface put me. This was before any of the color talks.)

The results of my testing are in and I threw away the big table I created as a test matrix because, with one exception, everything is based on the terminal program.

Conclusion

Users on Windows should always use Windows Terminal when running shells for the best experience.

Sadly, Windows users are going to install Git for windows which includes git-bash. Some of them will option for GUI Git, but others will just run git-bash from the start menu. We could provide a note showing where the appropriate bash is located to use Windows Terminal.

Test Results

Note: when I generically refer to win-bash I mean any of the systems that can be installed on windows to provide Unix commands. (git-bash, MSYS, Mingw, cygwin and Gow. I am going to ignore Gow, because it is primarily for running the commands and its bash doesn't work well as a terminal login shell.

The Terminals.

Most of the variation in color support and column (window resizing) support can be attributed to the terminal program running the shell.

Windows Console

• this is used by default when running cmd.exe or powershell, WSL
• It is possible to run all of the win-bash variants using Windows Console.
  • It takes extra work because they all have get installed with an entry added to the start menu that runs bash in mintty
  • It takes work to find the proper binary that can run in another terminal
• Color is supported
• Column sizing works most of the time. There are times when resizing the window narrower then running a command (putup -h) that the program thinks the window is much narrower that it actually is.

Mintty

• this is the default option for all win-bash variants
• Doesn't appear as tty in python
• Colors don't work
• Column sizing doesn't work and the program thinks there are 80 columns no matter terminal width
• Except when running cygwin bash, where everthing works just fine
  • cygwin is running the same version of mintty as the others

My go-to shell is git-bash so I tried to track this down and found references that said this is a problem with mintty, that it can't present as tty because it is actually using a hidden windows console to run bash in and they just using the mintty window as a mirror. Now that I found cygwin working, with the same version of mintty I will have to see if I can track down the difference.

Windows Terminal

• All shells mentioned can run in Windows Terminal
• Colors work
• Column sizing works

Mobaxterm

• I decided not to spend time testing because Windows Terminal will always be there and works for all.

[abravalheri]

Thank you very much @julie777 for the very thorough investigation!

I think we have 2 action points here:

• How to improve the current version of PyScaffold?
  We can add colorama as a conditional install_requires for Windows.
  (Adding dependencies is possible since v4.0, in v3.0 we had to be completely standalone for bootstrapping reasons. I imagine that the colorama dependency slipped under on our radar.)
• How we can improve future versions of PyScaffold?
  We should investigate the benefits of replacing our TUI layer with Rich (Use rich library to reduce codebase #344)

Other than that, I would consider the limitations of the terminal program under the optics of "graceful degradation"...
The users can still use PyScaffold and it will be fully functional, but the verbose output (which is not the default) will not be colorised.

[julie777]

@abravalheri
I am still wondering why cygwin works and git-bash doesn't when they are both using the same version of mintty. I am going to investigate further.

I will also install colorama and see if that works.

I don't understand why we would make colorama optional? Or do you mean conditional based on platform or env?

[abravalheri]

mean conditional based on platform or env?

☝️ this one! colorama is only used in Windows.