Add Ansible role for Windows DCAP CI testing

[ionutbalutoiu]

These changes address all the environment requirements
needed to do Azure-DCAP-Client CI testing on Windows.

This is a new set of Ansible tasks bundled into the file
windows/az-dcap-client/tasks/environment-setup.yml.

All the requirements are greatly documented in the issue #1320.

Fixes #1320

[ionutbalutoiu]

This PR is currently blocked by #1375. After that gets merged, I'll rebase the PR and we'll have only the Ansible Windows DCAP changes in the pull request diff.

Labeled this as blocked for now.

[ionutbalutoiu]

@CodeMonkeyLeet If you want to review these changes already, you can only take a look at the commit 21b970e since this is the entire codebase related to the this pull request.

[ionutbalutoiu]

@CodeMonkeyLeet, regarding the PR changes, I have 2 notes that I'd like to clarify with you:

• I choose to install the DCAP drivers via pnputil.exe tool here. I'm doing this since the pnputil.exe is already present on the system, while the devcon.exe needs to be fetched and setup (which is not that big of a deal, but I wanted a simple Ansible playbook).

  The drivers install succesfully, but I only give the full path to the sgx_base_dev.inf and sgx_dcap_dev.inf respectively.

  In the issue description, you pointed out the following devcon.exe usage:

  devcon.exe install sgx_base_dev.inf root\SgxLCDevice
  

  Looks like the second parameter is the driver store. I couldn't find how to pass this via pnputil.exe. Does pnputil.exe automatically detects the driver store, or should I use devcon.exe and explicitly specify that ?

• In the issue description, you mentioned:

  It would be nice if we can install to a target folder that does not include the nupkg version number, since they much be referenced for header include paths in the build project.
  

  Looking the NuGet CLI options, the parameter ExcludeVersion seems to be doing the job well.

  We install the nupkg's in C:\openenclave\prereqs\nuget using ExcludeVersion and we end up with the following nuget parent directory:

  openenclave
  └── prereqs
      └── nuget
          ├── curl
          ├── curl.redist
          ├── DCAP_Components
          ├── EnclaveCommonAPI
          ├── libssh2
          ├── libssh2.redist
          ├── Microsoft.Azure.DCAP.Client
          ├── openssl
          ├── openssl.redist
          ├── zlib
          └── zlib.redist
  
  

  as oposed to when ExcludeVersion is not used:

  openenclave
  └── prereqs
      └── nuget
          ├── curl.7.30.0.2
          ├── curl.redist.7.30.0.2
          ├── DCAP_Components.1.0.100.0
          ├── EnclaveCommonAPI.1.0.100.0
          ├── libssh2.1.4.3.1
          ├── libssh2.redist.1.4.3.1
          ├── Microsoft.Azure.DCAP.Client.1.0.0
          ├── openssl.1.0.1.21
          ├── openssl.redist.1.0.1.25
          ├── zlib.1.2.8.1
          └── zlib.redist.1.2.8.1
  

  Is this exactly what you had in mind ?

Regards,
Ionut

[ionutbalutoiu]

Quick update: The PR contains now only the Win DCAP CI Ansible changes.

[CodeMonkeyLeet]

Intel has since pushed out a newer version of the package at: http://registrationcenter-download.intel.com/akdlm/irc_nas/15025/Intel%20SGX%20DCAP%20for%20Windows%20v1.0.101.48697.exe

[ionutbalutoiu]

I'll update this.

[CodeMonkeyLeet]

@ionutbalutoiu, see replies below:

Does pnputil.exe automatically detects the driver store, or should I use devcon.exe and explicitly specify that ?

I don't think you need to explicitly specify it if you're using pnputil.exe. The .inf files themselves do contain an entry for desired device ID values, so pnputil.exe can probably pick that up. To double check this after installing the drivers:

1. Run devmgmt.msc (Device Manager) from the command line.
2. Under Software devices, for each device:
   1. Intel(R) Software Guard Extensions DCAP Components Device
   2. Intel(R) Software Guard Extensions Launch Configuration Service
3. Right-click and select Properties
4. Under the Details tab, select Matching device Id in the Property drop-down and check that the Value matches the expected ones:
   1. root\SgxLCDevice_DCAP
   2. root\SgxLCDevice

In the issue description, you mentioned:

It would be nice if we can install to a target folder that does not include the nupkg version number, since they much be referenced for header include paths in the build project.

Looking the NuGet CLI options, the parameter ExcludeVersion seems to be doing the job well.

Is this exactly what you had in mind ?

Yes, that looks like what I had in mind. If you look at the prototype for how we would like to consume that in the windows/dcap_support branch, the desire is that the hosts/CMakeLists.txt would not have to hardcode a version into the path reference for getting to the nuget header and libraries.

[CodeMonkeyLeet]

Generally looks good, but I'm not very familiar with Ansible. @johnkord might have to review that more closely.

@ionutbalutoiu, can you also check that using the updated Intel DCAP package works?

[CodeMonkeyLeet]

One additional note: the current scripts/ansible/README.md file doesn't mention any support for running playbooks on Windows. I assume it is supported, and if so, can we also update the instructions for how to run the playbook on Windows?

[ionutbalutoiu]

Generally looks good, but I'm not very familiar with Ansible. @johnkord might have to review that more closely.

@ionutbalutoiu, can you also check that using the updated Intel DCAP package works?

I noticed that the new DCAP package has only the DCAP_Components nupkg file included, while the previous one used to have both DCAP_Components and EnclaveCommonAPI nupkg files.

Do you know if the lack of EnclaveCommonAPI nupkg file is intentional ?

[ionutbalutoiu]

@ionutbalutoiu, see replies below:

Does pnputil.exe automatically detects the driver store, or should I use devcon.exe and explicitly specify that ?

I don't think you need to explicitly specify it if you're using pnputil.exe. The .inf files themselves do contain an entry for desired device ID values, so pnputil.exe can probably pick that up. To double check this after installing the drivers:

1. Run devmgmt.msc (Device Manager) from the command line.

2. Under Software devices, for each device:

   1. Intel(R) Software Guard Extensions DCAP Components Device
   2. Intel(R) Software Guard Extensions Launch Configuration Service

I cannot find any of these drivers listed under Software devices. I'm still debugging to see what's wrong with this.

3. Right-click and select Properties

4. Under the Details tab, select Matching device Id in the Property drop-down and check that the Value matches the expected ones:

   1. root\SgxLCDevice_DCAP
   2. root\SgxLCDevice

In the issue description, you mentioned:

It would be nice if we can install to a target folder that does not include the nupkg version number, since they much be referenced for header include paths in the build project.

Looking the NuGet CLI options, the parameter ExcludeVersion seems to be doing the job well.
Is this exactly what you had in mind ?

Yes, that looks like what I had in mind. If you look at the prototype for how we would like to consume that in the windows/dcap_support branch, the desire is that the hosts/CMakeLists.txt would not have to hardcode a version into the path reference for getting to the nuget header and libraries.

Awesome! Ansible installs everything to C:\openenclave\prereqs\nuget (this location is configurable via an Ansible variable).

Let me know if this location is good, or if I need to change it.

[ionutbalutoiu]

One additional note: the current scripts/ansible/README.md file doesn't mention any support for running playbooks on Windows. I assume it is supported, and if so, can we also update the instructions for how to run the playbook on Windows?

Yes. Windows is supported and tested as a targeted Ansible remote machine. I haven't tried to run Ansible from Windows directly though.

I only tried running Ansible from a Linux machine targeting both Linux and Windows remote machines. I'll try running from Windows directly and come back with my findings.

[ionutbalutoiu]

One additional note: the current scripts/ansible/README.md file doesn't mention any support for running playbooks on Windows. I assume it is supported, and if so, can we also update the instructions for how to run the playbook on Windows?

Yes. Windows is supported and tested as a targeted Ansible remote machine. I haven't tried to run Ansible from Windows directly though.

I only tried running Ansible from a Linux machine targeting both Linux and Windows remote machines. I'll try running from Windows directly and come back with my findings.

Hey @CodeMonkeyLeet,

It turns out that Ansible on Windows doesn't work right away. I tried to install the pip package and hit an issue described here.

People recommend using the Windows Linux Subsystem and have Ansible there. I think this is a good workaround. What do you think this ? I could add documentation about this.

[ionutbalutoiu]

@ionutbalutoiu, see replies below:

Does pnputil.exe automatically detects the driver store, or should I use devcon.exe and explicitly specify that ?

I don't think you need to explicitly specify it if you're using pnputil.exe. The .inf files themselves do contain an entry for desired device ID values, so pnputil.exe can probably pick that up. To double check this after installing the drivers:

1. Run devmgmt.msc (Device Manager) from the command line.

2. Under Software devices, for each device:

   1. Intel(R) Software Guard Extensions DCAP Components Device
   2. Intel(R) Software Guard Extensions Launch Configuration Service

I cannot find any of these drivers listed under Software devices. I'm still debugging to see what's wrong with this.

Yep. It looks like the installation method via pnputil.exe didn't get the drivers properly listed in Device Manager as you described.

Installing the drivers via devcon.exe has everything properly listed by Device Manager. I'll switch to devcon.exe install method and update the PR.

1. Right-click and select Properties

2. Under the Details tab, select Matching device Id in the Property drop-down and check that the Value matches the expected ones:

   1. root\SgxLCDevice_DCAP
   2. root\SgxLCDevice

[johnkord]

@ionutbalutoiu Could you modify the Windows maintainer document to include instructions on how to use this? I think this file would probably be appropriate: https://github.com/Microsoft/openenclave/blob/master/docs/GettingStartedDocs/GettingStarted.Windows.md

Perhaps we need to rename our documents again, so it's clear which documents are for maintainers and which are for SDK users (application developers)!

It's unfortunate that we either need WSL or to run this ansible playbook remotely to use ansible on Windows, but I don't think that's a complete blocker for maintainers to get a proper environment. I think it would be a major annoyance for SDK users (app devs), though. When we plan to support DCAP on Windows in a release, I imagine we'll want to include some of this logic in these ansible tasks for ACC deployments so that SDK users will be able to function without needing to run an ansible deployment. That's a work item I'll create now and we'll address when the time comes.

[ionutbalutoiu]

@ionutbalutoiu Could you modify the Windows maintainer document to include instructions on how to use this? I think this file would probably be appropriate: https://github.com/Microsoft/openenclave/blob/master/docs/GettingStartedDocs/GettingStarted.Windows.md

Perhaps we need to rename our documents again, so it's clear which documents are for maintainers and which are for SDK users (application developers)!

It's unfortunate that we either need WSL or to run this ansible playbook remotely to use ansible on Windows, but I don't think that's a complete blocker for maintainers to get a proper environment. I think it would be a major annoyance for SDK users (app devs), though. When we plan to support DCAP on Windows in a release, I imagine we'll want to include some of this logic in these ansible tasks for ACC deployments so that SDK users will be able to function without needing to run an ansible deployment. That's a work item I'll create now and we'll address when the time comes.

I'd create a new doc under docs/GettingStartedDocs called AnsibleGettingStarted.md.

I wouldn't include the Ansible usage steps in GettingStarted.Windows.md since after installing WSL, the steps to use Ansible are the same on both platforms.

What do you think about having a separate doc for Ansible ?

[johnkord]

@ionutbalutoiu That sounds fine to me, and if we want, we can refactor/reorganize the docs later when we plan to support DCAP on Windows! As long as the documentation exists somewhere easy to get to, I'm happy (for now) :)

[ionutbalutoiu]

@johnkord @CodeMonkeyLeet

I added to the pull request the documentation on how to use Ansible (the AnsibleGettingStarted.md doc), and I also added few more details on the Ansible README's.

If you want to execute the code from this pull request, do the following:

• Create a new Windows Server 2016 ACC VM
• Follow the AnsibleGettingStarted.md doc
• Run the Ansible role against the new Windows ACC VM via:

  ansible windows-agents -m import_role -a "name=windows/az-dcap-client tasks_from=environment-setup.yml"
  

There's a single remaining item on this pull request.

@CodeMonkeyLeet suggested using the new Intel DCAP installer (see conversation from here). I replied on this, but probably got lost in the whole GitHub conversation, please see my above comment.

[CodeMonkeyLeet]

@ionutbalutoiu

• Regarding the EnclaveCommonAPI package, that's an outstanding question I have with Intel they still haven't gotten back to me on, but based on the installation instructions I previously used, we don't reference it directly in the build on Windows ATM.

• It looks like we still don't have a good way to running ansible playbooks directly on Windows; WS2016 doesn't have WSL, so there's really no way to set up an ACC VM as a dev machine without a Linux server running Ansible to drive it? @johnkord Was that the only way to set up the box we gave to VS folks?

[ionutbalutoiu]

@CodeMonkeyLeet

Please, let me know if Intel reaches back with any response.

If you really wish to run Ansible under Windows releases without Linux subsystem support, there might be the following options:

1. Cygwin - provide functionality similar to a Linux distribution on Windows.
2. MSYS | MinGW - a collection of GNU utilities to allow building of applications and programs which depend on traditionally UNIX tools.

I tried only the first method, and I successfully run the Ansible tasks from this pull request, from a Windows 2016 box against my remote testing Windows 2016 ACC VM.

Digging further on this topic, it looks like Ansible won't work at all natively on Windows. So we have to use the Unix code under Windows in some way or another.

If you prefer any of the above methods, I could document these as well.

[johnkord]

@CodeMonkeyLeet The system I set up for the VS team didn't have any DCAP support, so it was very trivial to set up! I simply deployed a new Windows ACC VM and then installed the appropriate visual studio tools and clang-7, and everything else was already configured as part of the provision script that an ACC deployment runs.

[CodeMonkeyLeet]

@ionutbalutoiu I think that any requirement for Windows devs to have to spin up a custom Linux environment just to run Ansible so they can set up a Windows ACC dev box is going to be a poor experience. Honestly, it's faster for me to just read a set of instructions like the ones I gave you than jump through the hoops of setting up Ansible, and that's a red flag.

However, that is a distinct problem from what this PR addresses, so I've spun up #1499 to track one proposed alternative for managing the dev setup experience. For the purposes of this PR, it doesn't sound like we have remaining issues and should be able to proceed?

@johnkord, any additional blockers?

[johnkord]

@CodeMonkeyLeet Completely agree with you, Simon. Sounds good, let's proceed, and then address the enclave developer experience in the future. I like the plan to use docker for the development experience, and it hopefully can also help get us to a "reproducible build" scenario

[johnkord]

bors r+

[oe-bors]

👎 Rejected by too few approved reviews

[johnkord]

bors r+

[oe-bors]

Build succeeded

• continuous-integration/jenkins/branch