hook: Crash when using non-ASCII paths in command template

[pkel]

Using the config

hook:
  hooks:
    - event: album_imported
      command: beet convert -ay path:{album.path}

and this command

beet import -a -L path:/path/to/lib/Ария

I get an UnicodeDecodeError from hook.py line 50.

[jackwilsdon]

Hopefully #1980 fixes this. Could you try the latest master release and see if it's fixed?

[pkel]

Sadly not. The Error is now thrown in line 52.

Your new command.decode('utf8') is running on beet convert -ay path:{album.path}, isn't it? At which point is {album.path} resolved to ...\Ария\...?

I am new to python. This is all I can contribute without further explanations.

[jackwilsdon]

Would it be possible for you to post the exact stack trace (i.e. the full error) you get?

Thanks! 😄

[pkel]

Tagging:
    Ария - Игра с огнём
URL:
    http://musicbrainz.org/release/af43c159-86c6-4549-b669-e739f7717762
(Similarity: �[32;01m100.0%�[39;49;00m) �[37m(12" Vinyl, 1989, SU, Мелодия)�[39;49;00m
Traceback (most recent call last):
  File "/usr/local/bin/beet", line 9, in <module>
    load_entry_point('beets==1.3.18', 'console_scripts', 'beet')()
  File "/home/patrik/dev/beets/beets/ui/__init__.py", line 1249, in main
    _raw_main(args)
  File "/home/patrik/dev/beets/beets/ui/__init__.py", line 1239, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/home/patrik/dev/beets/beets/ui/commands.py", line 965, in import_func
    import_files(lib, paths, query)
  File "/home/patrik/dev/beets/beets/ui/commands.py", line 942, in import_files
    session.run()
  File "/home/patrik/dev/beets/beets/importer.py", line 320, in run
    pl.run_parallel(QUEUE_SIZE)
  File "/home/patrik/dev/beets/beets/util/pipeline.py", line 347, in run
    self.coro.send(msg)
  File "/home/patrik/dev/beets/beets/util/pipeline.py", line 160, in coro
    task = func(*(args + (task,)))
  File "/home/patrik/dev/beets/beets/importer.py", line 1400, in manipulate_files
    task.finalize(session)
  File "/home/patrik/dev/beets/beets/importer.py", line 543, in finalize
    self._emit_imported(session.lib)
  File "/home/patrik/dev/beets/beets/importer.py", line 569, in _emit_imported
    plugins.send('album_imported', lib=lib, album=self.album)
  File "/home/patrik/dev/beets/beets/plugins.py", line 458, in send
    result = handler(**arguments)
  File "/home/patrik/dev/beets/beets/plugins.py", line 123, in wrapper
    return func(*args, **kwargs)
  File "/home/patrik/dev/beets/beetsplug/hook.py", line 52, in hook_function
    **kwargs)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 22: ordinal not in range(128)

[jackwilsdon]

How peculiar. Do you think you could upload the album somewhere? Possibly mega.nz or Dropbox?

[pkel]

link by email

[sampsyo]

I'm pretty sure the problem is that album.path is a byte string. (All paths in beets are bytes—see Handling Paths.) So the problem arises for the same reason this little snippet crashes:

>>> u'{}'.format(u'caf\xe9'.encode('utf8'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)

I'm not 100% sure what the right solution is here—it's a deceptively tricky problem to support {album.path} and {album.title} at the same time, when one is bytes and the other is text!

But for the moment, @jackwilsdon, I don't think #1980 was necessary. The result of .get() on a configuration value should always be Unicode. (I just pushed 04bbded to make that obvious in the code.) I'm not 100% sure what calling .decode('utf8') on a Unicode string does, so we should probably roll that back.

[pkel]

So one workaround for me is changing

hook:
  hooks:
    - event: album_imported
      command: beet convert -ay path:{album.path}

to

hook:
  hooks:
    - event: album_imported
      command: beet convert -ay album:{album.title}

?

[sampsyo]

We do a similar thing in beets.logging.StrFormatLogger, but that only works for directly named arguments (plain {name}), not for more complicated lookups (like obj.field).

Perhaps it's possible to solve this with a Formatter subclass: https://docs.python.org/3/library/string.html#custom-string-formatting

Not that it's helpful, but a long-term solution would be the switch to pathlib (#1409). We could just provide a __format__ method on path objects that converts to a Unicode string sensibly.

[sampsyo]

@pkel, yeah, that would probably work. You might even try id:{album.id}.

[jackwilsdon]

@sampsyo I think you're right. .decode('utf8') does some unpredictable things on unicode strings. It seems to attempt to convert a unicode string back to ASCII before converting it to unicode again? 😆