Unicode issue in file path (username)/filename - daemon fails to start or downloads fail to work

[tzarebczan]

The Issue

If a user directory has unicode characters, the daemon fails to start. This particular user must have such characters in his Windows username, see logs below. The log file shows the username as ������, I'm checking with the user what his Username actually is, will update the issue later.

In some cases, the daemon works but the downloads folder does not. This results in trouble getting to the Settings tab (because it can't read the folder) and the directory needs to be manually set via daemon_settings.yml

To summarize, one of 3 things may happen if these characters exist in your windows username:
can't run daemon at all
can run daemon, but wallet fails to initialize and then crashes
can run daemon, but downloads fail to write to folder so it seems like you can't download

Steps to reproduce

1. Unicode character in Windows username
2. unicode characters in file name
3. Check log for errors.

Expected behaviour

Daemon starts up / fikes download

Actual behaviour

Daemon fails / files fail to save/access

System Configuration

• LBRY Daemon version: 0.13.1 / 0.14.2rc5
• LBRY App version:
• LBRY Installation ID:
• Operating system:

Anything Else

2017-07-21 23:33:12,217 INFO lbrynet.daemon.Daemon:241: Starting lbrynet-daemon
2017-07-21 23:33:12,358 INFO lbrynet.daemon.ExchangeRateManager:174: Starting exchange rate manager
2017-07-21 23:33:13,480 INFO lbrynet.daemon.Daemon:268: Platform: {"lbrynet_version": "0.14.2rc5", "platform": "Windows-8.1-6.3.9600", "os_system": "Windows", "python_version": "2.7.13", "os_release": "8.1", "lbryum_version": "3.1.4rc2", "ip": "xxxxx", "lbryschema_version": "0.0.8", "processor": "Intel64 Family 6 Model 69 Stepping 1, GenuineIntel", "build": "rc"}
2017-07-21 23:33:13,490 INFO lbrynet.daemon.Daemon:473: Loading databases
2017-07-21 23:33:13,500 INFO lbrynet.daemon.Daemon:537: Using lbryum wallet
2017-07-21 23:33:14,959 INFO lbrynet.core.Session:213: Set UPnP redirect for TCP port 3333
2017-07-21 23:33:15,118 INFO lbrynet.core.Session:224: Set UPnP redirect for UDP port 4444
2017-07-21 23:33:15,121 INFO lbrynet.core.Session:254: Starting DHT
2017-07-21 23:33:15,141 INFO lbrynet.dht.protocol:216: DHT listening on UDP 4444
2017-07-21 23:33:15,147 INFO lbrynet.core.RateLimiter:72: Starting <lbrynet.core.RateLimiter.RateLimiter object at 0x03FC91F0>
2017-07-21 23:33:15,147 INFO lbrynet.core.BlobManager:97: Setting up the DiskBlobManager. blob_dir: C:\Users\������\AppData\Roaming\lbrynet\blobfiles, db_file: C:\Users\������\AppData\Roaming\lbrynet\blobs.db
2017-07-21 23:33:15,151 ERROR lbrynet.daemon.DaemonControl:112: Failed to startup
Traceback (most recent call last):
File "site-packages\lbrynet\daemon\DaemonControl.py", line 109, in start_server_and_listen
FirstError: FirstError[#1, [Failure instance: Traceback: <class 'sqlite3.OperationalError'>: unable to open database file
threading.py:801:__bootstrap_inner
threading.py:754:run
site-packages\twisted_threads_threadworker.py:46:work
site-packages\twisted_threads_team.py:190:doWork
--- ---
site-packages\twisted\python\threadpool.py:246:inContext
site-packages\twisted\python\threadpool.py:262:
site-packages\twisted\python\context.py:118:callWithContext
site-packages\twisted\python\context.py:81:callWithContext
site-packages\twisted\enterprise\adbapi.py:464:_runInteraction
site-packages\twisted\enterprise\adbapi.py:36:init
site-packages\twisted\enterprise\adbapi.py:76:reconnect
site-packages\twisted\enterprise\adbapi.py:431:connect
]]
2017-07-21 23:33:15,153 INFO lbrynet.daemon.Daemon:402: Closing lbrynet session
2017-07-21 23:33:15,154 INFO lbrynet.daemon.Daemon:403: Status at time of shutdown: loading_wallet
2017-07-21 23:33:15,157 INFO lbrynet.core.Session:172: Shutting down <lbrynet.core.Session.Session object at 0x03FDACB0>
2017-07-21 23:33:15,158 INFO lbrynet.core.BlobAvailability:35: Stopping <lbrynet.core.BlobAvailability.BlobAvailabilityTracker object at 0x03F99A90>
2017-07-21 23:33:15,160 INFO lbrynet.core.RateLimiter:83: Stopping <lbrynet.core.RateLimiter.RateLimiter object at 0x03FC91F0>
2017-07-21 23:33:15,160 INFO lbrynet.core.client.DHTPeerFinder:28: Stopping <lbrynet.core.client.DHTPeerFinder.DHTPeerFinder object at 0x040EB2F0>
2017-07-21 23:33:15,161 INFO lbrynet.core.server.DHTHashAnnouncer:31: Stopping <lbrynet.core.server.DHTHashAnnouncer.DHTHashAnnouncer object at 0x040EB3D0>
2017-07-21 23:33:15,161 INFO lbrynet.core.Wallet:475: Stopping <lbrynet.core.Wallet.LBRYumWallet object at 0x03FDAB30>
2017-07-21 23:33:15,161 ERROR lbrynet.core.Wallet:541: Something went wrong during manage. Error message: 'NoneType' object has no attribute 'get_balance'
2017-07-21 23:33:15,167 ERROR lbrynet.core.Wallet:472: An error occurred stopping the wallet: Traceback (most recent call last):
File "site-packages\twisted\internet\defer.py", line 308, in addCallbacks

File "site-packages\twisted\internet\defer.py", line 651, in _runCallbacks

File "site-packages\lbrynet\core\Wallet.py", line 528, in

File "site-packages\twisted\internet\defer.py", line 1445, in unwindGenerator

--- ---
File "site-packages\twisted\internet\defer.py", line 1299, in _inlineCallbacks

File "site-packages\lbrynet\core\Wallet.py", line 558, in update_balance

File "site-packages\lbrynet\core\Wallet.py", line 1288, in _update_balance

File "site-packages\lbrynet\core\Wallet.py", line 1273, in _run_cmd_as_defer_succeed

File "site-packages\lbryum\commands.py", line 140, in func_wrapper

File "site-packages\lbryum\commands.py", line 358, in getbalance

exceptions.AttributeError: 'NoneType' object has no attribute 'get_balance'

2017-07-21 23:33:15,171 INFO lbrynet.core.BlobManager:101: Stopping the DiskBlobManager
2017-07-21 23:33:15,174 INFO lbrynet.core.Session:322: Unsetting upnp for <lbrynet.core.Session.Session object at 0x03FDACB0>
2017-07-21 23:33:15,177 INFO lbrynet.dht.protocol:468: Stopping DHT
2017-07-21 23:33:15,177 INFO lbrynet.dht.protocol:484: DHT stopped
2017-07-21 23:33:16,457 WARNING lbrynet.core.Session:333: UPnP redirect for TCP 3333 was removed by something else.
2017-07-21 23:33:16,542 WARNING lbrynet.core.Session:333: UPnP redirect for UDP 4444 was removed by something else.
2017-07-21 23:33:16,546 INFO lbrynet.daemon.Daemon:402: Closing lbrynet session
2017-07-21 23:33:16,548 INFO lbrynet.daemon.Daemon:403: Status at time of shutdown: loading_wallet
2017-07-21 23:33:16,549 INFO lbrynet.core.Session:172: Shutting down <lbrynet.core.Session.Session object at 0x03FDACB0>
2017-07-21 23:33:16,552 INFO lbrynet.core.BlobAvailability:35: Stopping <lbrynet.core.BlobAvailability.BlobAvailabilityTracker object at 0x03F99A90>
2017-07-21 23:33:16,552 INFO lbrynet.core.RateLimiter:83: Stopping <lbrynet.core.RateLimiter.RateLimiter object at 0x03FC91F0>
2017-07-21 23:33:16,552 INFO lbrynet.core.client.DHTPeerFinder:28: Stopping <lbrynet.core.client.DHTPeerFinder.DHTPeerFinder object at 0x040EB2F0>
2017-07-21 23:33:16,552 INFO lbrynet.core.server.DHTHashAnnouncer:31: Stopping <lbrynet.core.server.DHTHashAnnouncer.DHTHashAnnouncer object at 0x040EB3D0>
2017-07-21 23:33:16,553 INFO lbrynet.core.Wallet:475: Stopping <lbrynet.core.Wallet.LBRYumWallet object at 0x03FDAB30>
2017-07-21 23:33:16,553 ERROR lbrynet.core.Wallet:541: Something went wrong during manage. Error message: 'NoneType' object has no attribute 'get_balance'
2017-07-21 23:33:16,559 ERROR lbrynet.core.Wallet:472: An error occurred stopping the wallet: Traceback (most recent call last):
File "site-packages\twisted\internet\defer.py", line 308, in addCallbacks

File "site-packages\twisted\internet\defer.py", line 651, in _runCallbacks

File "site-packages\lbrynet\core\Wallet.py", line 528, in

File "site-packages\twisted\internet\defer.py", line 1445, in unwindGenerator

--- ---
File "site-packages\twisted\internet\defer.py", line 1299, in _inlineCallbacks

File "site-packages\lbrynet\core\Wallet.py", line 558, in update_balance

File "site-packages\lbrynet\core\Wallet.py", line 1288, in _update_balance

File "site-packages\lbrynet\core\Wallet.py", line 1273, in _run_cmd_as_defer_succeed

File "site-packages\lbryum\commands.py", line 140, in func_wrapper

File "site-packages\lbryum\commands.py", line 358, in getbalance

exceptions.AttributeError: 'NoneType' object has no attribute 'get_balance'

2017-07-21 23:33:16,562 INFO lbrynet.core.BlobManager:101: Stopping the DiskBlobManager
2017-07-21 23:33:16,565 INFO lbrynet.core.Session:322: Unsetting upnp for <lbrynet.core.Session.Session object at 0x03FDACB0>
2017-07-21 23:33:17,780 CRITICAL twisted:154: Unhandled error in Deferred:
2017-07-21 23:33:17,780 CRITICAL twisted:154:
Traceback (most recent call last):
File "site-packages\twisted\internet\defer.py", line 150, in maybeDeferred
File "site-packages\lbrynet\core\BlobManager.py", line 102, in stop
File "site-packages\twisted\enterprise\adbapi.py", line 397, in close
File "site-packages\twisted\enterprise\adbapi.py", line 405, in finalClose
File "site-packages\twisted\python\threadpool.py", line 274, in stop
File "site-packages\twisted_threads_team.py", line 226, in quit
File "site-packages\twisted_threads_convenience.py", line 35, in set
File "site-packages\twisted_threads_convenience.py", line 46, in check
AlreadyQuit

Screenshots

[tzarebczan]

Folder name is Богдан

[tzarebczan]

This is occurring more frequently as we onboard additional users. Got a handful of complaints with the same issue via helpdesk and slack.

[joemarchese]

First-timer here:
I think this is happening because sqlite3 expects utf-8. Using os.path.abspath() may fix it. I'm not really sure where, though. Maybe in lbry/lbrynet/conf.py

[kaykurokawa]

I think this might be the fix : #822

[tzarebczan]

I looked through some other logs, and another user had a slightly different error.
2017-07-05 21:42:21,788 INFO lbrynet.lbrynet_daemon.DaemonControl:92: Starting lbrynet-daemon from command line 2017-07-05 21:42:23,823 INFO lbrynet.lbrynet_daemon.DaemonServer:64: Using non-authenticated API 2017-07-05 21:42:23,826 INFO lbrynet.lbrynet_daemon.Daemon:254: Starting lbrynet-daemon 2017-07-05 21:42:24,006 INFO lbrynet.lbrynet_daemon.ExchangeRateManager:174: Starting exchange rate manager 2017-07-05 21:42:25,112 INFO lbrynet.lbrynet_daemon.Daemon:281: Platform: {"lbrynet_version": "0.13.1", "platform": "Windows-10-10.0.14393", "os_system": "Windows", "python_version": "2.7.13", "os_release": "10", "lbryum_version": "2.8.4", "ip": "178.48.40.185", "lbryschema_version": "0.0.7", "processor": "Intel64 Family 6 Model 94 Stepping 3, GenuineIntel", "build": "release"} 2017-07-05 21:42:25,117 INFO lbrynet.lbrynet_daemon.Daemon:488: Loading databases 2017-07-05 21:42:25,119 ERROR lbrynet.lbrynet_daemon.DaemonControl:128: Failed to startup Traceback (most recent call last): File "site-packages\lbrynet\lbrynet_daemon\DaemonControl.py", line 125, in start_server_and_listen File "site-packages\twisted\internet\defer.py", line 1297, in _inlineCallbacks File "site-packages\twisted\python\failure.py", line 389, in throwExceptionIntoGenerator File "site-packages\lbrynet\lbrynet_daemon\DaemonServer.py", line 49, in start File "site-packages\twisted\internet\defer.py", line 1297, in _inlineCallbacks File "site-packages\twisted\python\failure.py", line 389, in throwExceptionIntoGenerator File "site-packages\lbrynet\lbrynet_daemon\Daemon.py", line 261, in setup File "site-packages\twisted\python\threadpool.py", line 246, in inContext File "site-packages\twisted\python\threadpool.py", line 262, in <lambda> File "site-packages\twisted\python\context.py", line 118, in callWithContext File "site-packages\twisted\python\context.py", line 81, in callWithContext File "site-packages\lbrynet\lbrynet_daemon\Daemon.py", line 497, in _setup_data_directory UnicodeEncodeError: 'ascii' codec can't encode character u'\xf3' in position 10: ordinal not in range(128)

[tzarebczan]

Username: ćńóźżł
The new error is now:
` File "site-packages\twisted\python\context.py", line 118, in callWithContext
File "site-packages\twisted\python\context.py", line 81, in callWithContext
File "site-packages\lbrynet\daemon\Daemon.py", line 462, in _setup_data_directory
WindowsError: [Error 3] The system cannot find the path specified: 'C:\Users\cn\xf3zzl\Downloads'

[tzarebczan]

Per @kaykurokawa: the problem i think is not sqlite related, most likely will need to fix up this file https://github.com/lbryio/lbry/blob/master/lbrynet/winhelpers/knownpaths.py

[tzarebczan]

While troubleshooting with another user, I ran into yet a different scenario where LBRY daemon cannot even start up to the point where I can alter the data/lbryum/downloads paths via settings - the lbry folder is never created. This happens on Windows 10. On Windows 7, I get the lbry folders created and then can redirect the data/downloads/wallet dirs.

The user path is: C:\Users\Dellé\AppData

This error:
c:\Program Files (x86)\LBRY\resources\app\dist>lbrynet-daemon Traceback (most recent call last): File "site-packages\PyInstaller\loader\rthooks\pyi_rth_twisted.py", line 23, in <module> File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module File "site-packages\twisted\internet\default.py", line 56, in <module> File "site-packages\twisted\internet\default.py", line 50, in _getInstallFunction File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module File "site-packages\twisted\internet\selectreactor.py", line 18, in <module> File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module File "site-packages\twisted\internet\posixbase.py", line 18, in <module> File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module File "site-packages\twisted\internet\tcp.py", line 28, in <module> File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module File "site-packages\twisted\internet\_newtls.py", line 21, in <module> File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module File "site-packages\twisted\protocols\tls.py", line 41, in <module> File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module File "site-packages\OpenSSL\__init__.py", line 8, in <module> File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module File "site-packages\OpenSSL\crypto.py", line 12, in <module> File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module File "site-packages\cryptography\x509\__init__.py", line 9, in <module> File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module File "site-packages\cryptography\x509\base.py", line 16, in <module> File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module File "site-packages\cryptography\x509\extensions.py", line 13, in <module> File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module File "site-packages\asn1crypto\keys.py", line 22, in <module> File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module File "site-packages\asn1crypto\_elliptic_curve.py", line 51, in <module> File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module File "site-packages\asn1crypto\_int.py", line 56, in <module> File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module File "site-packages\asn1crypto\_perf\_big_num_ctypes.py", line 31, in <module> File "ctypes\util.py", line 53, in find_library File "ntpath.py", line 85, in join UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 11: ordinal not in range(128) Failed to execute script pyi_rth_twisted

[tzarebczan]

The current workaround (except for this particular case- #794 (comment)) is to manually set the working directories for LBRY in the default lbrynet directory (https://lbry.io/faq/lbry-directories).

You can manually set these directories in the daemon_settings.yml file. If this file does not exist in your lbrynet folder, you can use this one: https://drive.google.com/open?id=0BzKw_BSaBa_5ZXJzSDkyZXhVelU

This will setup your directories as follows or you can edit the file to change your own (but again, don't use any folders with non-English letters):
data_dir: c:\lbry\lbrynet
lbryum_wallet_dir: c:\lbry\lbryum
download_directory: c:\lbry\Downloads

[kennethreitz]

This should be relatively simple to solve.

Use Python 3 :D

I jest.

[kauffj]

@kennethreitz humor is the good natured side of a truth

[aa-novik]

I have the same problem with cyrillic usernames. I also had the same issue with Electrum, maybe this thread will help.

[MSFTserver]

i think this file needs a looking at maybe at a function to get language encoding user has and set encoding to that?
https://github.com/lbryio/lbry/blob/cb45d7cf8a1c7a2e9df4409729609a905f0a376b/lbrynet/conf.py

[MSFTserver]

i think this needs to change

def _win_path_to_bytes(path):
    """
    Encode Windows paths to string. appdirs.user_data_dir()
    on windows will return unicode path, unlike other platforms
    which returns string. This will cause problems
    because we use strings for filenames and combining them with
    os.path.join() will result in errors.
    """
    for encoding in ('ASCII', 'MBCS'):
        try:
            return path.encode(encoding)
        except (UnicodeEncodeError, LookupError):
            pass
    return path

[tzarebczan]

Come across 2 users who also had this issue, but it prevented from downloading/streaming content because it affected their Downloads directory - somehow the app started fine though.

The error:

1. 'utf8' codec can't decode byte 0xc2 in position 9: invalid continuation byte

[tzarebczan]

To summarize, one of 3 things may happen if these characters exist in your windows username:

1. can't run daemon at all
2. can run daemon, but wallet fails to initialize and then crashes
3. can run daemon, but downloads fail to write to folder so it seems like you can't download

[lyoshenka]

Here's what electrum did to solve this (or a similar) issue: spesmilo/electrum@6318bda

[tzarebczan]

Another example:

018-06-05 12:13:39,878 ERROR    lbrynet.lbry_file.client.EncryptedFileDownloader:158: Traceback (most recent call last):
  File "site-packages\lbrynet\lbry_file\client\EncryptedFileDownloader.py", line 155, in open_file
IOError: [Errno 22] invalid mode ('wb') or filename: 'C:\\Users\\Huni\\Downloads\\2018-\xec\xb2\xab-\xec\xb2\xa0\xea\xb5\xac\xeb\xa5\xbc-\xec\x9b\x83\xea\xb2\xa8\xeb\x9d\xbc-\xea\xb0\x9c\xeb\xa7\x89-\xec\x8b\x9c\xec\x9e\x91\xeb\xb6\x80\xed\x84\xb0-\xeb\xa0\x88\xec\xa0\x84\xeb\x93\x9c\xe3\x85\x8b\xe3\x85\x8b.mp4'

2018-06-05 12:13:39,887 WARNING  lbrynet.daemon.auth.server:219: error processing api request: Failed to open C:\Users\Huni\Downloads\2018-첫-철구를-웃겨라-개막-시작부터-레전드ㅋㅋ.mp4. Make sure you have permission to save files to that location.
traceback:   File "site-packages\twisted\internet\defer.py", line 1297, in _inlineCallbacks
    
  File "site-packages\twisted\python\failure.py", line 389, in throwExceptionIntoGenerator
    
  File "site-packages\lbrynet\daemon\Daemon.py", line 1687, in jsonrpc_get
    
  File "site-packages\twisted\internet\defer.py", line 1297, in _inlineCallbacks
    
  File "site-packages\twisted\python\failure.py", line 389, in throwExceptionIntoGenerator
    
  File "site-packages\lbrynet\file_manager\EncryptedFileDownloader.py", line 120, in _start
    
  File "site-packages\twisted\python\threadpool.py", line 246, in inContext
    
  File "site-packages\twisted\python\threadpool.py", line 262, in <lambda>
    
  File "site-packages\twisted\python\context.py", line 118, in callWithContext
    
  File "site-packages\twisted\python\context.py", line 81, in callWithContext
    
  File "site-packages\lbrynet\lbry_file\client\EncryptedFileDownloader.py", line 161, in open_file
    
exceptions.ValueError: Failed to open C:\Users\Huni\Downloads\2018-첫-철구를-웃겨라-개막-시작부터-레전드ㅋㅋ.mp4. Make sure you have permission to save files to that location.
2018-06-05 12:13:42,822 INFO     lbrynet.daemon.Daemon:1686: Already have lbry file but missing file in C:\Users\Huni\Downloads, rebuilding it
2018-06-05 12:13:42,823 WARNING  lbrynet.daemon.auth.server:219: error processing api request: CurrentlyStartingError
traceback:   File "site-packages\twisted\internet\defer.py", line 1299, in _inlineCallbacks
    
  File "site-packages\lbrynet\daemon\Daemon.py", line 1687, in jsonrpc_get
    
  File "site-packages\lbrynet\cryptstream\client\CryptStreamDownloader.py", line 87, in start
    
lbrynet.cryptstream.client.CryptStreamDownloader.CurrentlyStartingError: 
2018-06-05 12:14:00,612 INFO     lbrynet.daemon.Daemon:1686: Already have lbry file but missing file in C:\Users\Huni\Downloads, rebuilding it

[alyssaoc]

@tzarebczan please test after p3 integration

[alyssaoc]

@tzarebczan Can this issue be closed?

[tzarebczan]

Tested a bunch of the scenario above and I can happily say that this was indeed fixed by the python 3 upgrade! Woohoo!