Windows: Non-ascii username causes mach to fail

[PeterZhizhin]

Hello,

I am trying to build servo on Windows. My username in Windows is Петр (non-ascii) which causes builds to crash. I have manged to mitigate some issues while some are still unresolved. I still cannot build servo on my machine. Hope that somebody finds it helpful. I would also like to know if there is anything else that could be done to improve my building experience.

So, the username causes troubles when building.

1. I got the following error when trying to build from my directory:

Traceback (most recent call last):
  File "mach", line 103, in <module>
    main(sys.argv)
  File "mach", line 28, in main
    sys.path.insert(0, os.path.join(topdir, "python"))
  File "C:\Python27\lib\ntpath.py", line 85, in join
    result_path = result_path + p_path
UnicodeDecodeError: 'ascii' codec can't decode byte 0xcf in position 7: ordinal not in range(128)

My Project folder was C:\Users\Петр\Documents\Projects\servo which is non-ascii and non-ascii build paths are not supported (see #10002, #10003). This can be mitigated by moving the project to another directory.

2. I tried building with Python 3 because it has better unicode support and mach should support it. I ran mach by running py -3 mach. Here is the error:

Traceback (most recent call last):
  File ".\mach", line 58, in <module>
    from multiprocessing import forking
ImportError: cannot import name 'forking' from 'multiprocessing' (C:\Python37\lib\multiprocessing\__init__.py)

3. I tried to build in a folder that contains only ascii characters. The build still failed with the following error:

"C:\Python27\python.exe" "-m" "virtualenv" "-p" "C:\Python27\python.exe" "--system-site-packages" "C:\Users\peter\Documents\Projects\servo\python\_virtualenv2.7" failed with error code 1:
Output:
Error:
Traceback (most recent call last):
  File "C:\Python27\lib\runpy.py", line 163, in _run_module_as_main
    mod_name, _Error)
  File "C:\Python27\lib\runpy.py", line 111, in _get_module_details
    __import__(mod_name)  # Do not catch exceptions initializing package
  File "C:\Python27\lib\site-packages\virtualenv\__init__.py", line 3, in <module>
    from .run import cli_run
  File "C:\Python27\lib\site-packages\virtualenv\run\__init__.py", line 13, in <module>
    from .plugin.creators import CreatorSelector
  File "C:\Python27\lib\site-packages\virtualenv\run\plugin\creators.py", line 6, in <module>
    from virtualenv.create.via_global_ref.builtin.builtin_way import VirtualenvBuiltin
  File "C:\Python27\lib\site-packages\virtualenv\create\via_global_ref\builtin\builtin_way.py", line 7, in <module>
    from virtualenv.create.creator import Creator
  File "C:\Python27\lib\site-packages\virtualenv\create\creator.py", line 14, in <module>
    from virtualenv.discovery.cached_py_info import LogCmd
  File "C:\Python27\lib\site-packages\virtualenv\discovery\cached_py_info.py", line 26, in <module>
    _CACHE[Path(sys.executable)] = PythonInfo()
  File "C:\Python27\lib\site-packages\virtualenv\discovery\py_info.py", line 73, in __init__
    self.path = [u(i) for i in sys.path]
  File "C:\Python27\lib\site-packages\virtualenv\discovery\py_info.py", line 36, in u
    return v.decode("utf-8") if isinstance(v, bytes) else v
  File "C:\Python27\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xcf in position 9: invalid continuation byte

The error hinted that there was something with the sys.path variable. So, I tried inspecting it in:

>>> import sys
>>> sys.path
['', 'C:\\Users\\\xcf\xe5\xf2\xf0\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages', 'C:\\WINDOWS\\SYSTEM32\\python27.zip', 'C:\\Python27\\DLLs', 'C:\\Python27\\lib', 'C:\\Python27\\lib\\plat-win', 'C:\\Python27\\lib\\lib-tk', 'C:\\Python27', 'C:\\Python27\\lib\\site-packages']

OK, I removed this site-packages path (is was from a previous Python installation).

4. The next error that I got:

PS C:\Users\peter\Documents\Projects\servo> .\mach
Traceback (most recent call last):
  File "mach", line 103, in <module>
    main(sys.argv)
  File "mach", line 33, in main
    mach = mach_bootstrap.bootstrap(topdir)
  File "C:\Users\peter\Documents\Projects\servo\python\mach_bootstrap.py", line 291, in bootstrap
    _activate_virtualenv(topdir, is_firefox)
  File "C:\Users\peter\Documents\Projects\servo\python\mach_bootstrap.py", line 178, in _activate_virtualenv
    exec(compile(open(activate_path).read(), activate_path, 'exec'), dict(__file__=activate_path))
  File "C:\Users\peter\Documents\Projects\servo\python\_virtualenv2.7\Scripts\activate_this.py", line 21, in <module>
    os.environ["PATH"] = os.pathsep.join([bin_dir] + os.environ.get("PATH", "").split(os.pathsep))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xcf in position 9: ordinal not in range(128)

My $PATH system variable on Windows contains a lot of paths with the following prefix: C:\Users\Петр, so that's why it was crashing (see #20435). The variable contained a lot of scripts that were necessary for me and for Rust itself (for example, rustc.exe is located in C:\Users\Петр\.cargo\bin\rustc.exe).

At first, I got stuck here. But after complaining on Matrix, sebk advised to create a symlink. So, I did it:

mklink /D C:\Users\peter C:\Users\Петр

I then changed all instances of my user folder in $PATH to point to the symlink (including the one that contained %USERPROFILE% variable).

5. It worked at first. I tried to build. .\mach build --dev -p servo:

Error running mach:

    ['build', '--dev', '-p', 'servo']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

UnicodeEncodeError: 'ascii' codec can't encode characters in position 7-10: ordinal not in range(128)

  File "C:\Users\peter\Documents\Projects\servo\python\servo\build_commands.py", line 323, in build
    os.environ.update(eval(stdout.decode(encoding)))
  File "C:\Python27\lib\os.py", line 461, in update
    self[k] = dict[k]
  File "C:\Python27\lib\os.py", line 422, in __setitem__
    putenv(key, item)

The issue happens when it tries to decode the output of a Python snippet:

        if 'windows' in host:
            process = subprocess.Popen('("%s" %s > nul) && "python" -c "import os; print(repr(os.environ))"' %
                                       (os.path.join(vs_dirs['vcdir'], "Auxiliary", "Build", "vcvarsall.bat"), "x64"),
                                       stdout=subprocess.PIPE, shell=True)
            stdout, stderr = process.communicate()
            exitcode = process.wait()
            encoding = locale.getpreferredencoding()  # See https://stackoverflow.com/a/9228117
            if exitcode == 0:
                os.environ.update(eval(stdout.decode(encoding)))
            else:
                print("Failed to run vcvarsall. stderr:")
                print(stderr.decode(encoding))
                exit(1)

The code was added by #25365. I appears that it tried to copy my entire env (and decode that). Maybe we should output only necessary variables instead of spitting out all environment variables? I know that this would be a lot of variables to change and it could break things.

I am happy to help with the issue, just tell me what we can improve here.

[PeterZhizhin]

This kludge fixes the issue, cannot recommend for everyone (build_comands.py starting at line 323):

                environ_dict = eval(stdout.decode(encoding))
                valid_environ_dict = {}
                for key, value in environ_dict.items():
                    try:
                        value.decode('ascii')
                        valid_environ_dict[key] = value
                    except UnicodeEncodeError:
                        pass
                os.environ.update(valid_environ_dict)

Still looking for a better solution.

[PeterZhizhin]

I get the following error when building now:

[mozjs_sys 0.68.1] cargo:outdir=C:\Users\peter\Documents\Projects\servo\target\debug\build\mozjs_sys-594e8d1882776bb2\out\build
[mozjs_sys 0.68.1] [[ 'C:\Users\����\.cargo\git\checkouts\mozjs-fa11ffc7d4f1cc2d\9a6d8fc\mozjs'/js/src/configure -ot 'C:\Users\����\.cargo\git\checkouts\mozjs-fa11ffc7d4f1cc2d\9a6d8fc\mozjs'/js/src/configure.in ]] && touch 'C:\Users\����\.cargo\git\checkouts\mozjs-fa11ffc7d4f1cc2d\9a6d8fc\mozjs'/js/src/configure || true
[mozjs_sys 0.68.1] C:/Users/B19A~1/AppData/Local/Temp/make13324-1.sh: C:/Users/B19A~1/AppData/Local/Temp/make13324-1.sh: cannot execute binary file
[mozjs_sys 0.68.1] mozmake: *** [C:\Users\����\.cargo\git\checkouts\mozjs-fa11ffc7d4f1cc2d\9a6d8fc\makefile.cargo:190: maybe-configure] Error 126
[mozjs_sys 0.68.1] thread 'main' panicked at 'assertion failed: result.success()', C:\Users\Петр\.cargo\git\checkouts\mozjs-fa11ffc7d4f1cc2d\9a6d8fc\build.rs:177:5
[mozjs_sys 0.68.1] note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[PeterZhizhin]

This issue caused by Cargo home path set to C:\Users\Петр\.cargo and rustup path to C:\Users\Петр\.rustup.

Reinstalling Rust with RUSTUP_HOME=C:\Users\peter\.rustup and CARGO_HOME=C:\Users\peter\.cargo led to a couple of issues when building mozjs target:

1. It tried to find python2.7, but it couldn't. Adding a symlink mklink \D C:\Python27\python2.7.exe C:\Python27\python.exe solved it.
2. There is a line somewhere in mozjs configure scripts that resolves ~ path to C:\Users\Петр with os.path.expanduser. I looked at the ntpath.py system library and there was an override for this with $HOME variable. I set $HOME to point to C:\Users\peter. It resolved the issue.
3. For some reason, Cargo still adds C:\Users\Петр\.cargo to $PATH even when $CARGO_PATH is present. I hacked the configure.py script inside mozjs to work with my $PATH. Added the following line to C:\Users\peter\.cargo\git\checkouts\mozjs-fa11ffc7d4f1cc2d\9a6d8fc\mozjs\configure.py (after import os):

os.environ['PATH'] = ''.join(c for c in os.environ['PATH'] if ord(c) < 128).encode('ascii')

mozjs was built after all of that.

Hopefully these issues will be resolved after Mozilla moved to full Python 3 support.

[jdm]

Sorry there were so many issues, but I'm glad you were able to find workarounds.