Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

On Windows under Python 3, Colorama's ANSI color replacement doesn't work in cmd.exe #2607

Open
acastaner opened this issue Jun 21, 2017 · 23 comments

Comments

@acastaner
Copy link

@acastaner acastaner commented Jun 21, 2017

Problem

When running beets in a Windows 10 command-line terminal (cmd.exe) the output is wrong. It seems like there's a character encoding error. This is what it looks like (look at the last line where beets prompts for Skipping, Replace, etc) : https://imgur.com/a/b9K9x

E:\Music\_trier>beet -vv
user configuration: C:\Users\Arnaud\AppData\Roaming\beets\config.yaml
data directory: C:\Users\Arnaud\AppData\Roaming\beets
plugin paths:
Sending event: pluginload
inline: adding item field initial
inline: adding item field disc_and_track
library database: C:\Users\Arnaud\AppData\Roaming\beets\library.db
library directory: E:\Music
Sending event: library_opened
Usage:
  beet COMMAND [ARGS...]
  beet help COMMAND

Options:
  --format-item=FORMAT_ITEM
                        print with custom format
  --format-album=FORMAT_ALBUM
                        print with custom format
  -l LIBRARY, --library=LIBRARY
                        library database file to use
  -d DIRECTORY, --directory=DIRECTORY
                        destination music directory
  -v, --verbose         log more details (use twice for even more)
  -c CONFIG, --config=CONFIG
                        path to configuration file
  -h, --help            show this help message and exit

Commands:
  acousticbrainz    fetch metadata from AcousticBrainz
  clearart          remove images from file metadata
  config            show or edit the user configuration
  embedart          embed image files into file metadata
  extractart        extract an image from file metadata
  fetchart          download album art
  fields            show fields available for queries and format strings
  fingerprint       generate fingerprints for items without them
  ftintitle         move featured artists to the title field
  help (?)          give detailed help on a specific sub-command
  import (imp, im)  import new music
  lastgenre         fetch genres
  list (ls)         query the library
  modify (mod)      change metadata fields
  move (mv)         move or copy items
  remove (rm)       remove matching items from the library
  replaygain        analyze for ReplayGain
  stats             show statistics about the library or a query
  submit            submit Acoustid fingerprints
  update (upd, up)  update the library
  version           output version information
  write             write tag information to files
Sending event: cli_exit

Setup

  • OS: Windows 10 Professional
  • Python version: 3.6.1
  • beets version: 1.4.5
  • Turning off plugins made problem go away (yes/no): no

My configuration (output of beet config) is:

E:\Music\_trier>beet config
directory: E:\Music

plugins: inline acousticbrainz chroma embedart fetchart lastgenre replaygain ftintitle
threaded: yes
autotag: yes
art_filename: cover
asciify_paths: yes

clutter:
- .jpg
- .jpeg
- .png
- .sfv
- .nfo
- .m3u
- .m3u8
- Thumbs.DB
- .DS_Store
original_date: yes

format_item: $artist - $album - $title ($genre)

match:
    strong_rec_thresh: 0.04
    preferred:
        countries: [US, GB|UK, FR]
        media: [CD, Digital Media|File]
original_year: yes
item_fields:
    initial: albumartist[0].upper() + u'.'
    disc_and_track: u'%02i-%02i' % (disc, track) if disctotal > 1 else u'%02i' % (track)

paths:
    albumtype:soundtrack: ost/$album/$disc_and_track. $title
    savedir:arthur: _arthur/$album/$disc_and_track. $title
    savedir:radionova: nova/$album/$disc_and_track. $artist - $title
    savedir:world: world/$albumartist/$year - $album/$disc_and_track. $title
    savedir:electro: electro/$albumartist/$year - $album/$disc_and_track. $title
    savedir:classical: classical/$artist/$album/$disc_and_track. $title
    savedir:alt_rock: alt_rock/$albumartist/$year - $album/$disc_and_track. $title
    savedir:rock: rock/$albumartist/$year - $album/$disc_and_track. $title
    savedir:jazz: jazz/$albumartist/$year - $album/$disc_and_track. $title
    savedir:blues: blues/$albumartist/$year - $album/$disc_and_track. $title
    savedir:metal: metal/$albumartist/$year - $album/$disc_and_track. $title
    savedir:rap: rap/$albumartist/$year - $album/$disc_and_track. $title
    savedir:pop: pop/$albumartist/$year - $album/$disc_and_track. $title
    savedir:techno: techno/$albumartist/$year - $album/$disc_and_track. $title
    default: _trier/$albumartist/$year - $album/$disc_and_track. $title
    comp: $genre/$album/$track. $title

ui:
    color: yes
    colors:
        text_success: green
        text_warning: yellow
        text_error: red
        text_highlight: red
        text_highlight_minor: lightgray
        action_default: turquoise
        action: blue

import:
    write: yes
    copy: yes
    move: yes
    resume: ask
ftintitle:
    auto: yes
    drop: no
    format: feat. {0}
acousticbrainz:
    auto: yes
    force: no
chroma:
    auto: no
acoustid:
    apikey: REDACTED
embedart:
    auto: yes
    ifempty: yes
    maxwidth: 0
    compare_threshold: 0
    remove_art_file: no
fetchart:
    auto: yes
    cover_names: cover front art album folder
    sources: filesystem coverart itunes amazon albumart fanarttv
    minwidth: 0
    maxwidth: 0
    enforce_ratio: no
    cautious: no
    google_key: REDACTED
    google_engine: 001442825323518660753:hrh5ch1gjzm
    fanarttv_key: REDACTED
    store_source: no
lastgenre:
    auto: yes
    whitelist: yes
    min_weight: 10
    count: 1
    fallback:
    canonical: no
    source: album
    force: yes
    separator: ', '
    prefer_specific: no
lastfm:
    user: redacted
replaygain:
    backend: bs1770gain
    auto: yes
    overwrite: no
    targetlevel: 89
    r128: [Opus]
    chunk_at: 5000
    method: replaygain
copyartifacts:
    extensions: .cue .log
    print_ignored: yes
pathfields: {}
album_fields: {}
@sampsyo sampsyo added the needinfo label Jun 21, 2017
@sampsyo

This comment has been minimized.

Copy link
Member

@sampsyo sampsyo commented Jun 21, 2017

Hi! Those are console color control sequences. They're supposed to be automatically adapted into control commands for the Windows console.

Can you try running this command to check which platform beets believes it's running on?

python -c 'import sys; print(sys.platform)'

And this one to check whether Colorama, which provides colors on Windows, was successfully installed?

python -c 'import colorama; colorama.init()'

If Colorama is missing, you can install it with pip install colorama. But it should have been automatically installed when you installed beets, so that remains a mystery.

In the mean time, you can make the output more readable by setting color: no.

@acastaner

This comment has been minimized.

Copy link
Author

@acastaner acastaner commented Jun 21, 2017

Apparently cmd.exe doesn't like these commands :/

C:\Users\Arnaud>python -c 'import sys; print(sys.platform)'
  File "<string>", line 1
    'import
          ^
SyntaxError: EOL while scanning string literal

Disabling the colors works as a workaround though!

@sampsyo

This comment has been minimized.

Copy link
Member

@sampsyo sampsyo commented Jun 21, 2017

Sorry; I'm very far from being an expert in Windows command syntax. We just need a way to escape the spaces in that argument—maybe try double quotes instead of single quotes?

@acastaner

This comment has been minimized.

Copy link
Author

@acastaner acastaner commented Jun 21, 2017

No worries :)

The output of the first command is win32. The outpout of the second command is blank but colorama is apparently already installed:

C:\Users\Arnaud>pip install colorama
Requirement already satisfied: colorama in c:\users\arnaud\appdata\local\programs\python\python36-32\lib\site-packages

Note that within PowerShell there's no such error (it interprets the error correctly) ; but I tend to use cmd because PS has issues parsing the directory parameter (that's for another issue ;) )

@sampsyo

This comment has been minimized.

Copy link
Member

@sampsyo sampsyo commented Jun 21, 2017

That's strange. All this indicates that everything should be working.

Here's the relevant bit of the code:

# On Windows platforms, use colorama to support "ANSI" terminal colors.
if sys.platform == 'win32':
try:
import colorama
except ImportError:
pass
else:
colorama.init()

If you're interested in doing some more hardcore digging, the next step would be to sprinkle in some print() calls around there to check exactly what's going on. Something like this:

if sys.platform == 'win32':
    try:
        print('importing')
        import colorama
        print('imported')
    except ImportError:
        print('import error')
        pass
    else:
        print('initializing')
        colorama.init()
        print('initialized')

Seems silly, I know, but that would tell us exactly what your machine is doing.

@acastaner

This comment has been minimized.

Copy link
Author

@acastaner acastaner commented Jun 21, 2017

Doesn't seem silly at all, I'll try this and let you know.

@acastaner

This comment has been minimized.

Copy link
Author

@acastaner acastaner commented Jun 21, 2017

Tried the code snipped, and colorama seems to initialize fine?

E:\Music\blues\Muddy Waters>beet import "1989 - The Chess Box"
importing
imported
initializing
initialized

But then the rest of the import still shows the color control sequences.

@sampsyo

This comment has been minimized.

Copy link
Member

@sampsyo sampsyo commented Jun 21, 2017

Extremely mysterious! I took a look at the issues for Colorama and it sounds like there may be a problem with Colorama on Python 3 and writing raw bytes (which the only way we can control the output encoding). We probably have essentially the same issue as the one described here: tartley/colorama#125 (comment)

It looks like we'll need a different tactic. The problem, FWIW, is that this printing stanza:

beets/beets/ui/__init__.py

Lines 150 to 152 in 6185c6a

out = txt.encode(_out_encoding(), 'replace')
sys.stdout.buffer.write(out)
sys.stdout.buffer.flush()

is bypassing Colorama's translation by using sys.stdout.buffer to write bytes directly. Apparently, Colorama assumes that you only ever use sys.stdout directly. To control the encoding, we can't do that. (As an aside, I believe this is an important design flaw in Python 3—two steps forward, one step back.)

We'll need to look around for a way to coax Colorama into working on bytes on Python 3, in the same way as it does on Python 2. 😕 I'm not exactly sure how we'll manage this, but I'm marking this as a bug so we can look a little more closely.

@sampsyo sampsyo changed the title Beets output is displaying wrong on Windows 10 command-line On Windows under Python 3, Colorama's ANSI color replacement doesn't work in cmd.exe Jun 21, 2017
@sampsyo sampsyo added bug and removed needinfo labels Jun 21, 2017
@acastaner

This comment has been minimized.

Copy link
Author

@acastaner acastaner commented Jun 21, 2017

No problem, let me know if I can help further.

@RollingStar

This comment has been minimized.

Copy link
Contributor

@RollingStar RollingStar commented Jun 22, 2017

FWIW, I gave up on using cmd.exe and moved to Powershell.

@bleuge

This comment has been minimized.

Copy link

@bleuge bleuge commented Jul 3, 2017

Same here, moved to PS, unusable in standard Windows 10 commandline.

@sampsyo sampsyo added the python 3 label Jul 5, 2017
@acastaner

This comment has been minimized.

Copy link
Author

@acastaner acastaner commented Jul 12, 2017

I'm running into issues when using PS. It somehow can't find the specified directories:

PS C:\Users\Arnaud\Desktop> beet import '.\Overkill - 1989 - The Years of Decay (Megaforce) V0' --set savedir="metal"
error: no such file or directory: .\Overkill - 1989 - The Years of Decay (Megaforce) V0" --set savedir=metal

I'm not sure why it's doing this (and it's probably not related to beets) and that's why I tend to use cmd.exe instead.

@sampsyo

This comment has been minimized.

Copy link
Member

@sampsyo sampsyo commented Jul 12, 2017

Wow; that's pretty weird. It seems like single quotes should work as you'd expect… https://msdn.microsoft.com/en-us/powershell/reference/5.1/microsoft.powershell.core/about/about_quoting_rules

@jackwilsdon

This comment has been minimized.

Copy link
Member

@jackwilsdon jackwilsdon commented Dec 23, 2017

It sounds like it may be worth tracking tartley/colorama/issues/21, as that will fix this.

@sampsyo

This comment has been minimized.

Copy link
Member

@sampsyo sampsyo commented Dec 23, 2017

Good find, @jackwilsdon!

Yes, it does look like we either need to wait for that to get fixed, or we need to find a way to manually send bytes through the library.

@RollingStar

This comment has been minimized.

Copy link
Contributor

@RollingStar RollingStar commented Feb 10, 2018

FWIW, I can't get it to work anymore in Powershell either (maybe a Windows update broke it?). But it does work with Cmder Mini. http://cmder.net/

@ther0n

This comment has been minimized.

Copy link

@ther0n ther0n commented Feb 11, 2018

You can fix this issue in cmd.exe using ANSICON. Still doesn't fix the issue in powershell though.

http://softkube.com/blog/ansi-command-line-colors-under-windows

@matlads

This comment has been minimized.

Copy link

@matlads matlads commented Nov 2, 2018

Apparently using conEmu works well according to what this posted on discourse (see comment number 9):

https://discourse.beets.io/t/basic-usage/268/9

@tarrasque73

This comment has been minimized.

Copy link

@tarrasque73 tarrasque73 commented Mar 15, 2019

I can confirm that I have the same issue using the command prompt in windows 10 but not using ConEmu.

@jtmoon79

This comment has been minimized.

Copy link

@jtmoon79 jtmoon79 commented Mar 22, 2019

tl;dr same issue in powershell (not specific to cmd.exe), here's my setup.

Errant output

See the last part

PS> beet -v import ".\Quasi -- R&B Transmogrification"
no user configuration found at C:\Users\user1\AppData\Roaming\beets\config.yaml
data directory: C:\Users\user1\AppData\Roaming\beets
plugin paths:
Sending event: pluginload
library database: C:\Users\user1\AppData\Roaming\beets\library.db
library directory: C:\Users\user1\Music
Sending event: library_opened
Sending event: import_begin
state file could not be read: [Errno 2] No such file or directory: 'C:\\Users\\user1\\AppData\\Roaming\\beets\\state.pickle'
Sending event: import_task_created
Sending event: import_task_start
Looking up: C:\Temp\Quasi -- R&B Transmogrification
Tagging  -
No album ID found.
Search terms:  -
Album might be VA: True
Evaluating 0 candidates.

C:\Temp\Quasi -- R&B Transmogrification (14 items)
Sending event: before_choose_candidate
No matching release found for 14 tracks.
For help, see: http://beets.readthedocs.org/en/latest/faq.html#nomatch
�[36;01m[S]�[39;49;00mkip, �[34;01mU�[39;49;00mse as-is, as �[34;01mT�[39;49;00mracks, �[34;01mG�[39;49;00mroup albums, �[34;01mE�[39;49;00mnter search, enter �[34;01mI�[39;49;00md, a�[34;01mB�[39;49;00mort?

Using

Software versions, other stuff
  • python version 3.7.1

  • PowerShell 5.1.17763.316

  • Windows 10.0.17763.

  • python libraries

PS> pipenv graph
beets==1.4.7
  - colorama [required: Any, installed: 0.4.1]
  - jellyfish [required: Any, installed: 0.7.1]
  - munkres [required: Any, installed: 1.1.2]
  - musicbrainzngs [required: >=0.4, installed: 0.6]
  - mutagen [required: >=1.33, installed: 1.42.0]
  - pyyaml [required: Any, installed: 5.1]
  - six [required: >=1.9, installed: 1.12.0]
  - unidecode [required: Any, installed: 1.0.23]
  • other tests
PS> python -c 'import sys; print(sys.platform)'
win32
PS> python -c 'import colorama; colorama.init()'

(no error).


Possibly related to tartley/colorama/issues/48 (bettter captured by tartley/colorama/issues/79). Very likely other tartley/colorama issues, too.

@tman144566

This comment has been minimized.

Copy link

@tman144566 tman144566 commented Nov 3, 2019

I resolved this issue by enabling ANSI Terminal Control in the registry. I used Glenn Slayden solution.
https://superuser.com/questions/413073/windows-console-with-ansi-colors-handling

@KFFTFUFTUR

This comment has been minimized.

Copy link

@KFFTFUFTUR KFFTFUFTUR commented Apr 2, 2020

The problem seems to be in the print_ function that is beeing used to avoid exceptions by the default print function.

On windows we rely on colorama to convert ANSI color codes to win32 calls during encoding. Encoding the string directly seems to bypass that and break colorized output.

simply writing the text to stdout or even just passing it to the default print function would fix colors on windows however that would also throw encoding errors and break the encoding overwrite described in that function.

@sampsyo

This comment has been minimized.

Copy link
Member

@sampsyo sampsyo commented Apr 2, 2020

Yep, that’s a perfect summary. This thing about exceptions is still a sort of surprising downside of the Python 3 transition.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.