Suggestion: Specify the character set as utf-8 when opening the file in the file config.py, or use character set detection

[yinhedot]

Summary

Suggestion: Specify the character set as utf-8 when opening the file in the file config.py, or use character set detection.

Steps to reproduce

1. utf-8 char in config.toml:

[runner]
# 禁止掉 magic commands
magicEnabled = false

2. streamlit run xxx.py

3. get error

  File "c:\python38_64\lib\site-packages\streamlit\hashing.py", line 40, in <module>
    from streamlit.folder_black_list import FolderBlackList
  File "c:\python38_64\lib\site-packages\streamlit\folder_black_list.py", line 39, in <module>
    if config.get_option("global.developmentMode"):
  File "c:\python38_64\lib\site-packages\streamlit\config.py", line 104, in get_option
    config_options = get_config_options()
  File "c:\python38_64\lib\site-packages\streamlit\config.py", line 1055, in get_config_options
    file_contents = input.read()
UnicodeDecodeError: 'gbk' codec can't decode byte 0xad in position 6: illegal multibyte sequence

caused by config.py ,code snippet:

         with open(filename, "r") as input:
                file_contents = input.read()

Treatment method:

with open(filename, "r",encoding='utf-8') as input:
                file_contents = input.read()

[jroes]

This reminds me of #2615. Might be worth us auditing git grep open\( across the board for this kind of issue.

[yinhedot]

Good code practice, just like Golang source file must be encoded in UTF-8.

[jroes]

I did a little more research and found some interesting things:

• PEP 597 - proposal for defaulting to UTF-8
• From the proposal, I ran across how AWS is handling this with their CLI library is handling things.
• Another relevant discussion where some env vars re suggested, such as PYTHONUTF8

[kmcgrady]

Thanks for the report @yinhedot ! I think @jroes is right! We should just default to UTF-8.