Support building UWP app from mach

[jdm]

Our CI setup is dictated by https://github.com/servo/servo/blob/master/etc/taskcluster/decision_task.py . In particular, see the windows_unit and windows_arm64 functions which create tasks corresponding to https://tools.taskcluster.net/tasks/HtSwU2zzTdiC5wfpYIgLKg and https://tools.taskcluster.net/tasks/b6PRpaKmQ3GjcAkakQXXeg (chosen at random from https://treeherder.allizom.org/#/jobs?repo=servo-auto).

There are three main parts I expect to getting the UWP build to succeed:

• we need to install the ANGLE nuget package before building Servo, so the libraries can be linked as part of the build (https://docs.microsoft.com/en-us/nuget/tools/cli-ref-install)
• build libsimpleservo (same as https://github.com/servo/servo/wiki/Building-for-UWP)
• MSBuild ServoApp.sln /p:Configuration=Debug /p:Platform=x64 (example invocation to build the ServoApp solution in a particular configuration)

Once that's working it would be great to create nightly builds as well (see windows_nightly function in decision_task.py).

Update: we have nuget package installation working on CI in the arm64 build, and we're building the --uwp configuration. The missing piece is the msbuild execution, and if we add that to the mach build command automatically then CI gets it for free.

[highfive]

cc @wafflespeanut

[jdm]

I'm refining the scope of this work a bit. The CI work can happen separately; what I really would like is to make it as easy as possible to build the VS project as part of a single mach build command that also builds Servo.

I'm imagining that mach build -d --uwp should go through the existing build steps, and then once we're done packaging the DLLs we check if it's a uwp build and run the appropriate msbuild command. This will require determining the debug/release profile to use as well as the target platform based on the requested rust build target.

[angelortiz1007]

Observations of Current Build Steps:
While building servo (mach build -d --uwp), the build will not complete with a link time error as follows:
…
…
b" "ws2_32.lib" "advapi32.lib" "ws2_32.lib" "userenv.lib" "msvcrt.lib" "/DLL" "/IMPLIB:C:\git-mozilla\servo-angelortiz1007\target\debug\deps\simpleservo.dll.lib"
= note: lld-link: error: could not open libEGL.lib: no such file or directory
lld-link: error: could not open libEGL.lib:
no such file or directory

Resolution:
To resolve this link time error, the user is required to open the support\HoloLens\ServoApp.sln and attempt to build the x64 build type. It will also fail with the following msg:

1>c:\git-mozilla\servo-angelortiz1007\support\hololens\servoapp\browserpage.cpp(60): warning C4189: 'visible': local variable is initialized but not referenced
1>LINK : fatal error LNK1181: cannot open input file 'simpleservo.dll.lib'

This failure is due to the mach build -d --uwp link time error not building the simpleservo.dll.lib.

However, the required libegl.lib is then created/downloaded and found at:
support\hololens\packages\ANGLE.WindowsStore.Servo.2.1.13\bin\UAP\x64

Now the user can complete the mach build -d --uwp build where the simpleservo.dll.lib library is built.

Summary:
In order to automate a full build, the build steps would require the following when executing mach build -d/-r --uwp:

1. The nuget.exe package must be called to download the Angle package that puts the libegl.lib in support\hololens\packages\ANGLE.WindowsStore.Servo.2.1.13\bin\UAP\x64

2. complete the mach build -d/-r --uwp build where simpleservo.dll.lib is created for HoloLens/ServoApp.

3. Proceed to the ServoApp/HoloLens msbuild step.

During research it appears that nuget.exe can be used to download and place the libegl.lib into the packages directory for HoloLens using the following command:
nuget update ServoApp.sln
nuget restore ServoApp.sln

The above commands recreated the required packages directory for Angle without the need to open the ServoApp.sln and attempt a build.

After the above nuget commands were executed, the mach -build -r --uwp succeeded without any Servoapp build.

[jdm]

To automate the nuget work:

• add a dependency in our packages list for "nuget" with version "08-08-2019"
• this will cause a new nuget\08-08-2019 path to appear in .servo/msvc-dependencies/ when you next run mach build as https://servo-deps.s3.amazonaws.com/msvc-deps/nuget-08-08-2019.zip is downloaded and unzipped
• add a new extra_path entry for nuget (self.msvc_package_dir("nuget")) which will put nuget.exe in the PATH during the build command
• invoke subprocesses from python that use nuget as desired

[angelortiz1007]

I performed the 1st and 3rd bullet item above. After running mach build -d --uwp, I get the following error:

C:\git-mozilla\servo-angelortiz1007>mach build -d --uwp
Installing missing MSVC dependencies...
Getting parent_dir...
Getting zip_path and prepare_file...
Getting prepare_file and prepare_file...
Downloading https://servo-deps.s3.amazonaws.com/msvc-deps/nuget-08-08-2019.zip ...
Download failed (403): Forbidden - https://servo-deps.s3.amazonaws.com/msvc-deps/nuget-08-08-2019.zip
No Rust compiler binary available for this platform. Please see https://github.com/servo/servo/#prerequisites

I looked in the .servo/msvc-dependecies directory and see a file 08-08-2019.zip.part under nuget. It appears that there is something wrong with the downloading portion.

I suspect another file will need to be created/updated to specify nuget install ServoApp.sln.

Am I correct?

[jdm]

Oops. I didn't set the access permissions for the file correctly. It should work better now.

[angelortiz1007]

:)
A little better now. Download proceeds, but seems to terminate at around 97%. See error code below. This time the .servo\msvc-dependencies\nuget directory was blank.

C:\git-mozilla\servo-angelortiz1007>mach build -d --uwp
Installing missing MSVC dependencies...
Getting parent_dir...
Getting zip_path and prepare_file...
Getting prepare_file and prepare_file...
Downloading https://servo-deps.s3.amazonaws.com/msvc-deps/nuget-08-08-2019.zip ...
Downloading nuget-08-08-2019: 97.6%
Extracting nuget-08-08-2019...done
Getting extracted_path...
Getting os.rename...
Error running mach:

['build', '-d', '--uwp']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

WindowsError: [Error 2] The system cannot find the file specified

File "C:\git-mozilla\servo-angelortiz1007\python\servo\build_commands.py", line 238, in build
self.ensure_bootstrapped(target=target)
File "C:\git-mozilla\servo-angelortiz1007\python\servo\command_base.py", line 933, in ensure_bootstrapped
Registrar.dispatch("bootstrap", context=self.context)
File "C:\git-mozilla\servo-angelortiz1007\python_virtualenv\Lib\site-packages\mach\registrar.py", line 123, in dispatch
return self._run_command_handler(handler, context=context, **kwargs)
File "C:\git-mozilla\servo-angelortiz1007\python_virtualenv\Lib\site-packages\mach\registrar.py", line 90, in _run_command_handler
result = fn(**kwargs)
File "C:\git-mozilla\servo-angelortiz1007\python\servo\bootstrap_commands.py", line 47, in bootstrap
return bootstrap.bootstrap(self.context, force=force)
File "C:\git-mozilla\servo-angelortiz1007\python\servo\bootstrap.py", line 418, in bootstrap
return bootstrapper(context, force=force)
File "C:\git-mozilla\servo-angelortiz1007\python\servo\bootstrap.py", line 342, in windows_msvc
os.rename(extracted_path, package_dir(package))

C:\git-mozilla\servo-angelortiz1007>

[jdm]

Ok, I redid the archive to better fit the code's expectations about its contents. Try again?

[angelortiz1007]

Outstanding!!!

Success!

I now have .servo\msvc-dependencies\nuget\08-08-2019\nuget.exe

I got stumped initially and walked away and came back to it. After looking at the 403 message again I said there is something going on in the download.

I'll call it quits for the day knowing I passed this hurdle.:)

Thanks for the late night help!

[angelortiz1007]

Additional fine tuning may be needed.

I've noticed that "nuget.exe" also exists in ".servo\msvc-dependencies" in addition to ".servo\msvc-dependencies\nuget\08-08-2019"

[jdm]

That may be leftover from the previous failure.

[angelortiz1007]

I've created a python script to call nuget restore ServoApp.sln. My preliminary python script is:

def angle_root(target):
arch = {
"x86_64": "x64",
"x86": "X86",
"aarch64": "ARM64",
}
print(target)
angle_x64 = arch[target.split('-')[0]]
angle_default_path = path.join(os.getcwd(), "support\hololens\packages\ANGLE.WindowsStore.Servo.2.1.13\bin\UAP\")
angle_default_path += angle_x64

# Nuget executable command
nuget_exe = path.join(os.getcwd(), ".servo\\msvc-dependencies\\nuget\\08-08-2019\\nuget.exe ")
nuget_action = "restore "
nuget_app = path.join(os.getcwd(), "support\\hololens\\ServoApp.sln")
nuget_command = nuget_exe + nuget_action + nuget_app

if os.path.exists(angle_default_path):
    print("Angle Found...")
else:
    print("Angle not found.  Performing nuget to restore Angle package")
    os.system(nuget_command)

return angle_default_path

The python function angle_root() may need to be called from build_commands.py around line 290...

        # Don't want to mix non-UWP libraries with vendored UWP libraries.
        if "gstreamer" in env['LIB']:
            print("Found existing GStreamer library path in LIB. Please remove it.")
            sys.exit(1)

        # Override any existing GStreamer installation with the vendored libraries.
        env["GSTREAMER_1_0_ROOT_" + arch['gst']] = path.join(
            self.msvc_package_dir("gstreamer-uwp"), arch['gst_root']
        )

[angelortiz1007]

That may be leftover from the previous failure.

I thought too, but I deleted it and reran mach build -d --uwp and after the download nuget.exe was observed.

[angelortiz1007]

I'm refining the scope of this work a bit. The CI work can happen separately; what I really would like is to make it as easy as possible to build the VS project as part of a single mach build command that also builds Servo.
I'm imagining that mach build -d --uwp should go through the existing build steps, and then once we're done packaging the DLLs we check if it's a uwp build and run the appropriate msbuild command. This will require determining the debug/release profile to use as well as the target platform based on the requested rust build target.

Just to confirm and document:
Within the build_commands.py, the function below is responsible for passing the Configuration dev build type (Release/Debug) and the Platform build type win_arm64 (x86 vs arm64).

def build(self, release=False, dev=False, jobs=None, params=None,
no_package=False, verbose=False, very_verbose=False,
target=None, android=False, magicleap=False, libsimpleservo=False, uwp=False,
features=None, win_arm64=False, **kwargs):

dev specifies (Debug/Release) msbuild -p Configuration type (if dev == false implies Release, if dev == true implies a Debug)

win_arm64 specifies x64 (target = "x86-64-pc-windows-msvc) or AARCH64 (target = "aarch64-pc-windows-msvc)

[angelortiz1007]

Reference Link to resolve msbuild time error of missing certificate

https://www.oipapio.com/question-1459134

snipet .....
Same issue I met these days, And I finally successfully solved with the help of this post.
Actually we must ensure we had create a test-used .pfx as the developer of UWP ,and followed words shared some helpful solutions I thought:
Create a test certificate with VS:
In Visual Studio, from Solution Explorer, open Package.appxmanifest
In the App Manifest Designer, choose the Packaging tab, and then choose the Choose Certificate button.
In the Choose Certificate dialog box, expand the Configure Certificate list, and then choose Create test certificate.
In the Create test certificate dialog box, input your info and click the OK button.(Above step description are coming from this ↓)
Create a certificate with MakeCert,Pvk2Pfx and Certutil tools
MDSN : How to create an app package signing certificate
Hope it can help u!

[angelortiz1007]

got my python scripts working well. However, when I run msbuild for debug and x64, msbuild is building arm64 as well.

Here is my command:
msbuild /m:4 /p:project=ServoApp .\support\hololens\servoapp.sln /p:SolutionDir=c:\git-mozilla\angelortiz1007-2\support\hololens /p:Configuration="Debug" /p:Platform="x64" /p:AdditionalIncludePaths=c:\git-mozilla\angelortiz1007-2\target\debug

The following is another article that discusses this same issue:
https://stackoverflow.com/questions/36789559/msbuild-always-build-all-platforms-even-when-specific-one-provided

[angelortiz1007]

got my python scripts working well. However, when I run msbuild for debug and x64, msbuild is building arm64 as well.
Here is my command:
msbuild /m:4 /p:project=ServoApp .\support\hololens\servoapp.sln /p:SolutionDir=c:\git-mozilla\angelortiz1007-2\support\hololens /p:Configuration="Debug" /p:Platform="x64" /p:AdditionalIncludePaths=c:\git-mozilla\angelortiz1007-2\target\debug
The following is another article that discusses this same issue:
https://stackoverflow.com/questions/36789559/msbuild-always-build-all-platforms-even-when-specific-one-provided

The suggestion from the link above appears to work. The following msbuild command produces only the release|x64 target.
msbuild /m:4 /p:project=ServoApp .\support\hololens\servoapp.sln /p:SolutionDir=c:\git-mozilla\angelortiz1007-2\support\hololens /p:Configuration="Release" /p:Platform="x64" /property:AppxBundle=Always;AppxBundlePlatforms="x64"

Changing "Release" to "Debug" also works.