Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Suggestion: set encoding explicitly when reading settings assets #962

Closed
semio opened this issue Dec 24, 2021 · 0 comments · Fixed by #966
Closed

Suggestion: set encoding explicitly when reading settings assets #962

semio opened this issue Dec 24, 2021 · 0 comments · Fixed by #966
Assignees
Labels
bug Something isn't working

Comments

@semio
Copy link

semio commented Dec 24, 2021

Overview

I tried to run frictionless on Windows but I got following error:

~\Anaconda3\envs\gapminder\lib\site-packages\frictionless\helpers.py in <module>
     20 from urllib.parse import urlparse, parse_qs
     21 from _thread import RLock  # type: ignore
---> 22 from . import settings
     23
     24

~\Anaconda3\envs\gapminder\lib\site-packages\frictionless\settings.py in <module>
     24 REPORT_PROFILE = json.loads(read_asset("profiles", "report.json"))
     25 STATUS_PROFILE = json.loads(read_asset("profiles", "status.json"))
---> 26 SCHEMA_PROFILE = json.loads(read_asset("profiles", "schema", "general.json"))
     27 RESOURCE_PROFILE = json.loads(read_asset("profiles", "resource", "general.json"))
     28 TABULAR_RESOURCE_PROFILE = json.loads(read_asset("profiles", "resource", "tabular.json"))

~\Anaconda3\envs\gapminder\lib\site-packages\frictionless\settings.py in read_asset(*paths)
     11     dirname = os.path.dirname(__file__)
     12     with open(os.path.join(dirname, "assets", *paths)) as file:
---> 13         return file.read().strip()
     14
     15

UnicodeDecodeError: 'gbk' codec can't decode byte 0xac in position 7809: illegal multibyte sequence

This is because the default encoding for open() is platform dependent. UTF-8 mode is the default on Linux but it's not on Windows. So python tries to read the asset files with my local encoding.

I can fix this by setting PYTHONUTF8 environment variable, but I think that it could be better if you set encoding in open() to UTF8 because you have control over the assets files.


Please preserve this line to notify @roll (lead of this repository)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

2 participants