UnicodeEncodeError: 'utf-8' codec can't encode character '\udce9' in position 14: surrogates not allowed #5648

kalon33 · 2025-02-26T14:39:01Z

Problem

Running this command in verbose (-vv) mode:

$ $ ./bin/beet -vv import -gC /media/nicolas/musique/
user configuration: /home/nicolas/.config/beets/config.yaml
data directory: /home/nicolas/.config/beets
plugin paths: 
lyrics: Disabling google source: no API key configured.
fetchart: google: Disabling art source due to missing key
fetchart: lastfm: Disabling art source due to missing key
Sending event: pluginload
library database: /media/nicolas/musique/musiclibrary.db
library directory: /media/nicolas/musique
Sending event: library_opened
Sending event: import_begin
state file could not be read: [Errno 2] No such file or directory: '/home/nicolas/.config/beets/state.pickle'
ignoring '.directory' due to ignore rule '.*'
ignoring '.folder.png' due to ignore rule '.*'
Traceback (most recent call last):
  File "/home/nicolas/beets/lib/python3.12/site-packages/beets/util/__init__.py", line 413, in bytestring_path
    return str_path.encode(_fsencoding())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'utf-8' codec can't encode character '\udce9' in position 14: surrogates not allowed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/nicolas/beets/./bin/beet", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/nicolas/beets/lib/python3.12/site-packages/beets/ui/__init__.py", line 1870, in main
    _raw_main(args)
  File "/home/nicolas/beets/lib/python3.12/site-packages/beets/ui/__init__.py", line 1849, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/home/nicolas/beets/lib/python3.12/site-packages/beets/ui/commands.py", line 1398, in import_func
    import_files(lib, paths, query)
  File "/home/nicolas/beets/lib/python3.12/site-packages/beets/ui/commands.py", line 1329, in import_files
    session.run()
  File "/home/nicolas/beets/lib/python3.12/site-packages/beets/importer.py", line 360, in run
    pl.run_parallel(QUEUE_SIZE)
  File "/home/nicolas/beets/lib/python3.12/site-packages/beets/util/pipeline.py", line 446, in run_parallel
    raise exc_info[1].with_traceback(exc_info[2])
  File "/home/nicolas/beets/lib/python3.12/site-packages/beets/util/pipeline.py", line 260, in run
    msg = next(self.coro)
          ^^^^^^^^^^^^^^^
  File "/home/nicolas/beets/lib/python3.12/site-packages/beets/importer.py", line 1444, in read_tasks
    yield from task_factory.tasks()
  File "/home/nicolas/beets/lib/python3.12/site-packages/beets/importer.py", line 1259, in tasks
    for dirs, paths in self.paths():
  File "/home/nicolas/beets/lib/python3.12/site-packages/beets/importer.py", line 1309, in paths
    for dirs, paths in albums_in_dir(self.toppath):
  File "/home/nicolas/beets/lib/python3.12/site-packages/beets/importer.py", line 1770, in albums_in_dir
    for root, dirs, files in sorted_walk(
  File "/home/nicolas/beets/lib/python3.12/site-packages/beets/util/__init__.py", line 229, in sorted_walk
    base = bytestring_path(str_base)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nicolas/beets/lib/python3.12/site-packages/beets/util/__init__.py", line 415, in bytestring_path
    return str_path.encode("utf-8")
           ^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'utf-8' codec can't encode character '\udce9' in position 14: surrogates not allowed

Led to this problem:

I can't use beets to tag my music library.

Setup

OS: Ubuntu 24.04.2 LTS
Python version: 3.12
beets version: 2.2.0
Turning off plugins made problem go away (yes/no): no

My configuration (output of beet config) is:

lyrics:
    bing_lang_from: []
    auto: yes
    synced: yes
    bing_client_secret: REDACTED
    bing_lang_to:
    google_API_key: REDACTED
    google_engine_ID: REDACTED
    genius_api_key: REDACTED
    fallback:
    force: no
    local: no
    sources:
    - google
    - genius
    - tekstowo
    - lrclib
    dist_thresh: 0.1
directory: /media/nicolas/musique
# --------------- Main ---------------

library: /media/nicolas/musique/musiclibrary.db

import:
    copy: no

# --------------- Plugins ---------------

plugins: discogs lyrics autobpm chroma fetchart lastgenre lyrics replaygain
discogs:
    apikey: REDACTED
    apisecret: REDACTED
    tokenfile: discogs_token.json
    source_weight: 0.5
    user_token: REDACTED
    separator: ', '
    index_tracks: no
    append_style_genre: no
fetchart:
    auto: yes
    minwidth: 0
    maxwidth: 0
    quality: 0
    max_filesize: 0
    enforce_ratio: no
    cautious: no
    cover_names:
    - cover
    - front
    - art
    - album
    - folder
    sources:
    - filesystem
    - coverart
    - itunes
    - amazon
    - albumart
    - cover_art_url
    store_source: no
    high_resolution: no
    deinterlace: no
    cover_format:
    google_key: REDACTED
    google_engine: 001442825323518660753:hrh5ch1gjzm
    fanarttv_key: REDACTED
    lastfm_key: REDACTED
lastgenre:
    whitelist: yes
    min_weight: 10
    count: 1
    fallback:
    canonical: no
    source: album
    force: yes
    auto: yes
    separator: ', '
    prefer_specific: no
    title_case: yes
replaygain:
    overwrite: no
    auto: yes
    backend: command
    threads: 12
    parallel_on_import: no
    per_disc: no
    peak: 'true'
    targetlevel: 89
    r128: [Opus]
    r128_targetlevel: 84
    command: ''
    noclip: yes
chroma:
    auto: yes
autobpm:
    auto: yes
    overwrite: no
    beat_track_kwargs: {}

Thanks for your help!

The text was updated successfully, but these errors were encountered:

maybe fix beetbox#5648

aszaw · 2025-02-28T00:22:41Z

this could be due to the processed file or directory having surrogate chars (chars beyond xFFFF) when going through the walk, will test the commit on my branch later

m4caque · 2025-02-28T16:19:37Z

I'm seeing the same issue as well. It seems to be when printing candidate results when running an import:

Evaluating 10 candidates.
Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/beets/util/__init__.py", line 415, in bytestring_path
    return str_path.encode(_fsencoding())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'utf-8' codec can't encode character '\udce1' in position 6: surrogates not allowed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/beet", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/lib/python3.12/site-packages/beets/ui/__init__.py", line 1870, in main
    _raw_main(args)
  File "/usr/lib/python3.12/site-packages/beets/ui/__init__.py", line 1849, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/usr/lib/python3.12/site-packages/beets/ui/commands.py", line 1398, in import_func
    import_files(lib, paths, query)
  File "/usr/lib/python3.12/site-packages/beets/ui/commands.py", line 1329, in import_files
    session.run()
  File "/usr/lib/python3.12/site-packages/beets/importer.py", line 360, in run
    pl.run_parallel(QUEUE_SIZE)
  File "/usr/lib/python3.12/site-packages/beets/util/pipeline.py", line 446, in run_parallel
    raise exc_info[1].with_traceback(exc_info[2])
  File "/usr/lib/python3.12/site-packages/beets/util/pipeline.py", line 260, in run
    msg = next(self.coro)
          ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/beets/importer.py", line 1444, in read_tasks
    yield from task_factory.tasks()
  File "/usr/lib/python3.12/site-packages/beets/importer.py", line 1259, in tasks
    for dirs, paths in self.paths():
                       ^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/beets/importer.py", line 1309, in paths
    for dirs, paths in albums_in_dir(self.toppath):
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/beets/importer.py", line 1770, in albums_in_dir
    for root, dirs, files in sorted_walk(
                             ^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/beets/util/__init__.py", line 262, in sorted_walk
    yield from sorted_walk(cur, ignore_bytes, ignore_hidden, logger)
  File "/usr/lib/python3.12/site-packages/beets/util/__init__.py", line 231, in sorted_walk
    base = bytestring_path(str_base)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/beets/util/__init__.py", line 417, in bytestring_path
    return str_path.encode("utf-8")
           ^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'utf-8' codec can't encode character '\udce1' in position 6: surrogates not allowed

Fixes beetbox#5648

bwoodford · 2025-03-09T18:58:26Z

I was also encountering this issue. I tried out the fix that @aszaw put in place, but the error was still getting propagated. I rearranged the exception handlers and now the import is working as expected.

snejus · 2025-03-10T07:35:21Z

I have a feeling this issue may have been caused by #5611

snejus · 2025-03-10T07:50:18Z

#5662 could you check whether this helps?

aszaw added a commit to aszaw/beets that referenced this issue Feb 27, 2025

Update __init__.py

ab574b5

maybe fix beetbox#5648

bwoodford added a commit to bwoodford/beets that referenced this issue Mar 9, 2025

Update __init__.py

9d1d7ea

Fixes beetbox#5648

snejus self-assigned this Mar 10, 2025

snejus linked a pull request Mar 10, 2025 that will close this issue

Use os.fsencode to encode bytes #5662

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UnicodeEncodeError: 'utf-8' codec can't encode character '\udce9' in position 14: surrogates not allowed #5648

UnicodeEncodeError: 'utf-8' codec can't encode character '\udce9' in position 14: surrogates not allowed #5648

kalon33 commented Feb 26, 2025 •

edited

Loading

aszaw commented Feb 28, 2025

m4caque commented Feb 28, 2025

bwoodford commented Mar 9, 2025

snejus commented Mar 10, 2025

snejus commented Mar 10, 2025

UnicodeEncodeError: 'utf-8' codec can't encode character '\udce9' in position 14: surrogates not allowed #5648

UnicodeEncodeError: 'utf-8' codec can't encode character '\udce9' in position 14: surrogates not allowed #5648

Comments

kalon33 commented Feb 26, 2025 • edited Loading

Problem

Setup

aszaw commented Feb 28, 2025

m4caque commented Feb 28, 2025

bwoodford commented Mar 9, 2025

snejus commented Mar 10, 2025

snejus commented Mar 10, 2025

kalon33 commented Feb 26, 2025 •

edited

Loading