Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash on import with UTF-8 names under Mac OS X #1248

Closed
claunia opened this issue Jan 20, 2015 · 11 comments
Closed

Crash on import with UTF-8 names under Mac OS X #1248

claunia opened this issue Jan 20, 2015 · 11 comments

Comments

@claunia
Copy link

claunia commented Jan 20, 2015

Trying to import any file with unicode names, or inside a unicode-named folder, crashes in Mac OS X.

Tested with beets 1.3.8 and 1.3.10, under Mac OS X 10.9.5 and 10.10.0.

Traceback (most recent call last):
File "/usr/local/bin/beet", line 9, in
load_entry_point('beets==1.3.10', 'console_scripts', 'beet')()
File "/Library/Python/2.7/site-packages/beets/ui/init.py", line 945, in main
_raw_main(args)
File "/Library/Python/2.7/site-packages/beets/ui/init.py", line 935, in _raw_main
subcommand.func(lib, suboptions, subargs)
File "/Library/Python/2.7/site-packages/beets/ui/commands.py", line 873, in import_func
import_files(lib, paths, query)
File "/Library/Python/2.7/site-packages/beets/ui/commands.py", line 845, in import_files
session.run()
File "/Library/Python/2.7/site-packages/beets/importer.py", line 305, in run
pl.run_parallel(QUEUE_SIZE)
File "/Library/Python/2.7/site-packages/beets/util/pipeline.py", line 299, in run
out = self.coro.send(msg)
File "/Library/Python/2.7/site-packages/beets/util/pipeline.py", line 181, in coro
func(*(args + (task,)))
File "/Library/Python/2.7/site-packages/beets/importer.py", line 1268, in plugin_stage
func(session, task)
File "/Library/Python/2.7/site-packages/beetsplug/keyfinder.py", line 52, in imported
self.find_key(task.items)
File "/Library/Python/2.7/site-packages/beetsplug/keyfinder.py", line 70, in find_key
.format(key, util.displayable_path(item.path)))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 104: ordinal not in range(128)
@claunia
Copy link
Author

claunia commented Jan 20, 2015

Disabling keyfinder plugin stops crashing.

Keyfinder configuration:
keyfinder:
bin: /Applications/KeyFinder.app/Contents/MacOS/KeyFinder
auto: yes
overwrite: no

@sampsyo
Copy link
Member

sampsyo commented Jan 20, 2015

Hi! Thank you for reporting. I believe this should be fixed in git due to some recent refactoring of the logging machinery. If you get a chance, can you confirm that this is working now?

@sampsyo sampsyo closed this as completed Jan 20, 2015
@claunia
Copy link
Author

claunia commented Jan 20, 2015

Unfortunately using git master just changes the error:

Traceback (most recent call last):
File "/usr/local/bin/beet", line 9, in
load_entry_point('beets==1.3.11', 'console_scripts', 'beet')()
File "/Library/Python/2.7/site-packages/beets/ui/init.py", line 947, in main
_raw_main(args)
File "/Library/Python/2.7/site-packages/beets/ui/init.py", line 937, in raw_main
subcommand.func(lib, suboptions, subargs)
File "/Library/Python/2.7/site-packages/beets/ui/commands.py", line 866, in import_func
import_files(lib, paths, query)
File "/Library/Python/2.7/site-packages/beets/ui/commands.py", line 843, in import_files
session.run()
File "/Library/Python/2.7/site-packages/beets/importer.py", line 314, in run
pl.run_parallel(QUEUE_SIZE)
File "/Library/Python/2.7/site-packages/beets/util/pipeline.py", line 299, in run
out = self.coro.send(msg)
File "/Library/Python/2.7/site-packages/beets/util/pipeline.py", line 181, in coro
func((args + (task,)))
File "/Library/Python/2.7/site-packages/beets/importer.py", line 1332, in plugin_stage
func(session, task)
File "/Library/Python/2.7/site-packages/beets/plugins.py", line 119, in wrapper
result = func(_args, **kwargs)
File "/Library/Python/2.7/site-packages/beetsplug/keyfinder.py", line 48, in imported
self.find_key(task.items)
File "/Library/Python/2.7/site-packages/beetsplug/keyfinder.py", line 68, in find_key
item.store()
File "/Library/Python/2.7/site-packages/beets/library.py", line 239, in store
super(LibModel, self).store()
File "/Library/Python/2.7/site-packages/beets/dbcore/db.py", line 354, in store
tx.mutate(query, subvars)
File "/Library/Python/2.7/site-packages/beets/dbcore/db.py", line 655, in mutate
cursor = self.db._connection().execute(statement, subvals)
sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

@sampsyo
Copy link
Member

sampsyo commented Jan 20, 2015

Wow, a completely different bug! Thanks for reporting. Looking into this now.

sampsyo added a commit that referenced this issue Jan 20, 2015
@sampsyo
keyfinder: Better output parsing (#1248)
f617d16 
We were being sloppy about bytes output from the process. Also, it seems like
the tools outputs the path also, so it's necessary to break on whitespace to
actually get the key name.
@sampsyo
Copy link
Member

sampsyo commented Jan 20, 2015

I took a closer look and found several glaring issues in the keyfinder plugin. 😱 They should be fixed now.

@claunia
Copy link
Author

claunia commented Jan 20, 2015

Good, updating to master of masters and testing, stay tuned 😛

@claunia
Copy link
Author

claunia commented Jan 20, 2015

I have a new bug, but at least, now, it doesn't stop importing:

Logged from file keyfinder.py, line 63
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 467, in format
    s = self._fmt % record.__dict__
  File "/Library/Python/2.7/site-packages/beets/logging.py", line 46, in __str__
    return self.msg.format(*self.args, **self.kwargs)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xcc in position 139: ordinal not in range(128)
Logged from file keyfinder.py, line 63

sampsyo added a commit that referenced this issue Jan 21, 2015
@sampsyo
Handle encoding misfortunes in logging (#1248)
9b83cef 
The beets loggers now, as proposed in #1044, do their utmost to avoid crashing
when logged values are not valid Unicode or Unicode-convertible. This came up
urgently when a `CalledProcessError` could not be logged with `'u{}'.format()`
(!!!) so we needed to some manner of conversion, but it should be possible to
log paths without `displayable_path`. In fact, it may be time to retire
`displayable_path` for the most part.
@sampsyo
Copy link
Member

sampsyo commented Jan 21, 2015

Thank you again, and my apologies for the number of bugs you're uncovering in this (apparently neglected) plugin. This latest one should also be fixed now.

@claunia
Copy link
Author

claunia commented Jan 21, 2015

No need to apologize, it's been my pleasure to help a fellow developer uncover and kill nasty bugs on a software I do like 👍

@darthanubis
Copy link

Traceback (most recent call last):
File "/usr/bin/beet", line 9, in
load_entry_point('beets==1.3.8', 'console_scripts', 'beet')()
File "/usr/share/beets/beets/ui/init.py", line 964, in main
_raw_main(args)
File "/usr/share/beets/beets/ui/init.py", line 954, in _raw_main
subcommand.func(lib, suboptions, subargs)
File "/usr/share/beets/beets/ui/commands.py", line 883, in import_func
import_files(lib, paths, query)
File "/usr/share/beets/beets/ui/commands.py", line 855, in import_files
session.run()
File "/usr/share/beets/beets/importer.py", line 297, in run
pl.run_parallel(QUEUE_SIZE)
File "/usr/share/beets/beets/util/pipeline.py", line 299, in run
out = self.coro.send(msg)
File "/usr/share/beets/beets/util/pipeline.py", line 158, in coro
task = func(*(args + (task,)))
File "/usr/share/beets/beets/importer.py", line 1148, in user_query
resolve_duplicates(session, task)
File "/usr/share/beets/beets/importer.py", line 1158, in resolve_duplicates
found_duplicates = task.find_duplicates(session.lib)
File "/usr/share/beets/beets/importer.py", line 570, in find_duplicates
album_paths = set(i.path for i in album.items())
File "/usr/share/beets/beets/library.py", line 795, in items
return self._db.items(dbcore.MatchQuery('album_id', self.id))
File "/usr/share/beets/beets/library.py", line 1103, in items
return self._fetch(Item, query, sort)
File "/usr/share/beets/beets/library.py", line 1086, in _fetch
model_cls, query, sort
File "/usr/share/beets/beets/dbcore/db.py", line 758, in _fetch
rows = tx.query(sql, subvals)
File "/usr/share/beets/beets/dbcore/db.py", line 599, in query
return cursor.fetchall()
sqlite3.OperationalError: Could not decode to UTF-8 column 'lyrics' with text '[Intro:]
It's the Diamonds baby
[Notorious B.I.G.] "Everybody move ya body" [x8]

beets crashes on import when getting to Insomnia: The Erick Sermon Compilation Album The Wixtons
The lyrics plugin then crashes attempting to read the lyrcis.

beet lyrics Insomnia: The Erick Sermon Compilation Album The Wixtons
Traceback (most recent call last):
File "/usr/bin/beet", line 9, in
load_entry_point('beets==1.3.8', 'console_scripts', 'beet')()
File "/usr/share/beets/beets/ui/init.py", line 964, in main
_raw_main(args)
File "/usr/share/beets/beets/ui/init.py", line 954, in _raw_main
subcommand.func(lib, suboptions, subargs)
File "/usr/share/beets/beetsplug/lyrics.py", line 492, in func
for item in lib.items(ui.decargs(args)):
File "/usr/share/beets/beets/library.py", line 1103, in items
return self._fetch(Item, query, sort)
File "/usr/share/beets/beets/library.py", line 1086, in _fetch
model_cls, query, sort
File "/usr/share/beets/beets/dbcore/db.py", line 758, in _fetch
rows = tx.query(sql, subvals)
File "/usr/share/beets/beets/dbcore/db.py", line 599, in query
return cursor.fetchall()
sqlite3.OperationalError: Could not decode to UTF-8 column 'lyrics' with text '[Intro:]
It's the Diamonds baby
[Notorious B.I.G.] "Everybody move ya body" [x8]

Jah Boogie:
Here comes the Jah changin' faces spoiled rotten
A.

Not sure if this is related to this bug, as I didn't want to create a duplicate bug report.

@sampsyo
Copy link
Member

sampsyo commented May 15, 2015

@darthanubis This is a separate bug. Can you provide a copy of your database for testing (associated with a new issue)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@sampsyo @claunia @darthanubis