Skip to content
No description, website, or topics provided.
Branch: master
Clone or download
Pull request Compare This branch is 5 commits ahead, 347 commits behind mozilla-releng:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
ci
userdata
.taskcluster.yml
readme.md

readme.md

Open Cloud Config (OCC)

Docker pulls Deploy status

Issues

To file bugs, raise issues or request features, please use: b.m.o/infra-ops/occ.

Usage

OCC is a small, fast, lightweight tool for creating Windows cloud (they don't really have to be in a cloud though) instances with a specific configuration in a repeatable, source controlled manner. Think of it as Puppet or Chef, without all the orchestration. There isn't even anything to install. It's implemented in a few powershell scripts, hosted in this repository.

OCC has no dependencies other than powershell so you shouldn't have to install anything on the instances where you want it to run. In EC2 for example, you can provide a single command in your userdata (and a great big json manifest out on the web somewhere), in order to build a specific instance configuration at startup.

Powershell Desired State Configuration (DSC) is used as the provider.

Which manifest to run is currently determined by the ssh key associated with the instance in AWS EC2 (key naming convention). There are currently manifests for:

Running the following command at an elevated powershell prompt (or providing it as EC2 userdata) will start OCC on an instance:

Invoke-Expression (New-Object Net.WebClient).DownloadString('https://raw.githubusercontent.com/mozilla-releng/OpenCloudConfig/master/userdata/rundsc.ps1')

Instance configuration is defined in json format and currently includes implementations for these instance configuration mechanisms (most source parameters are expected to be a URL):

  • DirectoryCreate: Create an empty folder or validate that it already exists

    example:

    {
      "ComponentName": "TempDirectory",
      "ComponentType": "DirectoryCreate",
      "Path": "C:\\Temp"
    }
    
  • DirectoryDelete: Delete a folder and all contents or validate that it does not exist

    example:

    {
      "ComponentName": "TempDirectory",
      "ComponentType": "DirectoryDelete",
      "Path": "C:\\Temp"
    }
    
  • DirectoryCopy: Copy a folder and all its contents or validate that destination is identical to source (including contents)

    example:

    {
      "ComponentName": "AliceDirectory",
      "ComponentType": "DirectoryCopy",
      "Source": "C:\\Users\\Bob",
      "Target": "C:\\Users\\Alice"
    }
    
  • CommandRun: Run a command from the cmd (COMSPEC) command prompt. Provides an optional mechanism for first performing a validation step to check if the command should be run

    example:

    {
      "ComponentName": "pip-upgrade-virtualenv",
      "ComponentType": "CommandRun",
      "Command": "C:\\Python27\\python.exe",
      "Arguments": [
        "-m",
        "pip",
        "install",
        "--upgrade",
        "virtualenv==15.0.1"
      ],
      "DependsOn": [
        {
          "ComponentType": "ExeInstall",
          "ComponentName": "PythonSetup"
        }
      ],
      "Validate": {
        "CommandsReturn": [
          {
            "Command": "C:\\Python27\\python.exe",
            "Arguments": [
              "-m",
              "pip",
              "show",
              "virtualenv"
            ],
            "Match": "Version: 15.0.1"
          }
        ]
      }
    }
    
  • FileDownload: Download a file from source or validate that a file with the same name exists at destination

    example:

    {
      "ComponentName": "Win32ToolToolManifest",
      "ComponentType": "ChecksumFileDownload",
      "Source": "https://hg.mozilla.org/mozilla-central/raw-file/tip/browser/config/tooltool-manifests/win32/releng.manifest",
      "Target": "C:\\Windows\\Temp\\releng.manifest.win32.tt"
    }
    
  • ChecksumFileDownload: Download a file from source or validate that a file with the same name and SHA1 signature exists at destination

    example:

    {
      "ComponentName": "VisualStudio2015AdminDeployment",
      "ComponentType": "ChecksumFileDownload",
      "Source": "https://raw.githubusercontent.com/mozilla-releng/OpenCloudConfig/master/userdata/Configuration/VisualStudio2015/AdminDeployment.xml",
      "Target": "C:\\Windows\\Temp\\VisualStudio2015AdminDeployment.xml"
    }
    
  • SymbolicLink: Create a symbolic link (file or directory) or validate that it already exists

    example:

    {
      "ComponentName": "Home",
      "ComponentType": "SymbolicLink",
      "Target": "C:\\Users",
      "Link": "C:\\home"
    }
    
  • ExeInstall: Install an executable or validate that it has already been installed (using optional validation commands)

    example:

    {
      "ComponentName": "VisualStudio2015Community",
      "ComponentType": "ExeInstall",
      "Url": "http://download.microsoft.com/download/0/B/C/0BC321A4-013F-479C-84E6-4A2F90B11269/vs_community.exe",
      "Arguments": [
        "/Passive",
        "/NoRestart",
        "/AdminFile",
        "C:\\Windows\\Temp\\VisualStudio2015AdminDeployment.xml",
        "/Log",
        "C:\\log\\VisualStudio2015CommunityInstall.log"
      ],
      "DependsOn": [
        {
          "ComponentType": "ChecksumFileDownload",
          "ComponentName": "VisualStudio2015AdminDeployment"
        }
      ],
      "Validate": {
        "PathsExist": [
          "C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\Common7\\IDE\\devenv.exe",
          "C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin\\mspdb140.dll",
          "C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin\\amd64\\mspdb140.dll"
        ]
      }
    }
    
  • MsiInstall: Install an MSI or validate that it has already been installed (using product identifier)

    example:

    {
      "ComponentName": "BinScope",
      "ComponentType": "MsiInstall",
      "Comment": "https://dxr.mozilla.org/mozilla-central/search?q=BinScope&redirect=false&case=false",
      "Url": "https://download.microsoft.com/download/2/6/A/26AAA1DE-D060-4246-93B5-7D7C877E4F8F/BinScopeSetup.msi",
      "Name": "SDL BinScope",
      "ProductId": "B137EB8C-FA6C-4DA7-95F0-A9B6FFE67A64"
    }
    
  • WindowsFeatureInstall: Install a Windows feature or validate that it is already installed

    example:

    {
      "ComponentName": "NET-Framework-Core",
      "ComponentType": "WindowsFeatureInstall",
      "Name": "NET-Framework-Core",
      "DependsOn": [
        {
          "ComponentType": "ServiceControl",
          "ComponentName": "Start-wuauserv"
        }
      ]
    }
    
  • ZipInstall: Extract a compressed archive from source to destination or validate that archive contents already exist at destination

  • ServiceControl: Set service startup type and trigger the expected service state or validate that the startup type and service are already in the expected state

    example:

    {
      "ComponentName": "Start-wuauserv",
      "ComponentType": "ServiceControl",
      "Comment": "Required by NET-Framework-Core",
      "Name": "wuauserv",
      "StartupType": "Manual",
      "State": "Running"
    }
    
  • EnvironmentVariableSet: Set an environment variable or validate that it has been set to the provided value

    example:

    {
      "ComponentName": "env-MOZILLABUILD",
      "ComponentType": "EnvironmentVariableSet",
      "DependsOn": [
        {
          "ComponentType": "ExeInstall",
          "ComponentName": "MozillaBuildSetup"
        }
      ],
      "Name": "MOZILLABUILD",
      "Value": "C:\\mozilla-build",
      "Target": "Machine"
    }
    
  • EnvironmentVariableUniqueAppend: Append one or more values to a collection environment variable delimited by semicolon or validate that the variable already contains the required value(s)

    example:

    {
      "ComponentName": "env-PATH",
      "ComponentType": "EnvironmentVariableUniqueAppend",
      "DependsOn": [
        {
          "ComponentType": "ExeInstall",
          "ComponentName": "PythonSetup"
        }
      ],
      "Name": "PATH",
      "Values": [
        "C:\\Python27\\python",
        "C:\\Python27\\python\\Scripts"
      ],
      "Target": "Machine"
    }
    
  • EnvironmentVariableUniquePrepend: Prepend one or more values to a collection environment variable delimited by semicolon or validate that the variable already contains the required value(s)

    example:

    {
      "ComponentName": "env-PATH",
      "ComponentType": "EnvironmentVariableUniquePrepend",
      "DependsOn": [
        {
          "ComponentType": "ExeInstall",
          "ComponentName": "MozillaBuildSetup"
        }
      ],
      "Name": "PATH",
      "Values": [
        "C:\\mozilla-build\\7zip",
        "C:\\mozilla-build\\info-zip",
        "C:\\mozilla-build\\kdiff3",
        "C:\\mozilla-build\\moztools-x64\\bin",
        "C:\\mozilla-build\\mozmake",
        "C:\\mozilla-build\\msys\\bin",
        "C:\\mozilla-build\\msys\\local\\bin",
        "C:\\mozilla-build\\nsis-3.0b1",
        "C:\\mozilla-build\\nsis-2.46u",
        "C:\\mozilla-build\\python",
        "C:\\mozilla-build\\python\\Scripts",
        "C:\\mozilla-build\\upx391w",
        "C:\\mozilla-build\\wget",
        "C:\\mozilla-build\\yasm"
      ],
      "Target": "Machine"
    }
    
  • RegistryKeySet: Set a registry key or validate that the key already exists

    example:

    {
      "ComponentName": "reg-WindowsErrorReportingLocalDumps",
      "ComponentType": "RegistryKeySet",
      "Comment": "",
      "Key": "HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\Windows\\Windows Error Reporting",
      "ValueName": "LocalDumps"
    }
    
  • RegistryValueSet: Set a registry key and value or validate that the key already exists and contains the specified value

    example:

    {
      "ComponentName": "reg-WindowsErrorReportingDontShowUI",
      "ComponentType": "RegistryValueSet",
      "Comment": "",
      "Key": "HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\Windows\\Windows Error Reporting",
      "ValueName": "DontShowUI",
      "ValueType": "Dword",
      "ValueData": "0x00000001",
      "Hex": true
    }
    
  • FirewallRule: Set a firewall rule or validate that the rule already exists

    example:

    {
      "ComponentName": "LiveLog-Get",
      "ComponentType": "FirewallRule",
      "Protocol": "TCP",
      "LocalPort": 60022,
      "Direction": "Inbound",
      "Action": "Allow"
    }
    
  • ReplaceInFile: Replace text in a file (optionally using a powershell expression or variable)

    example:

    {
      "ComponentName": "SetNxlogAggregator",
      "ComponentType": "ReplaceInFile",
      "Path": "C:\\Program Files (x86)\\nxlog\\conf\\nxlog.conf",
      "Match": "DatacenterToken",
      "Replace": "$env:datacenter"
    }
    

OpenCloudConfig CI

When a commit is pushed to the mozilla-releng/OpenCloudConfig master branch, a set of tasks will be run.

Previous tasks

To see previous tasks that have run, visit the commits page and look for green ticks, or heaven forbid, red crosses.

Implementation

Those tasks are generated from the /.taskclsuter.yml file in the root of this repository via a taskcluster-github integration.

AMI building

They will build Amazon Machine Images for a set of taskcluster worker types, which are defined in the /userdata/Manifest directory of this git repository. Each json manifest file represents one specific worker type.

Updating live worker type definitions

In addition to rebuilding the AMIs, the CI will also update live production AWS worker type definitions to use the new images.

Defining which worker types to update

Tasks can be filtered based on the syntax of the commit message of the master branch commit head. To update all OCC worker types, include deploy: all in your commit message. To rebuild a selection of worker types, provide a space delimited list in your deploy message, e.g. deploy: <workertype1> <workertype2> ... in your commit message.

This is usually most easily accomplished using two messages in the commit command; the first the regular commit message, and the second for the deployment instruction, e.g.:

git commit -m 'Something important for GPU beta worker types' \
           -m 'deploy: gecko-t-win10-64-gpu-b gecko-t-win7-32-gpu-b'

Note - a task will still be created for each possible workertype, however it will immediately exit when it runs, if it is not included in the deploy syntax.

Also note, if you do not include deploy syntax in your commit, nothing will get deployed. This is to safeguard automatic deploys from accidental pushes.

Redeploying without changes

If you find yourself needing to retrigger a deployment, or deploy to an additional worker type, but without making source code changes, this can be accomplished with an empty commit as follows:

git commit --allow-empty \
           -m "Doing this because I'm special." \
           -m 'deploy: gecko-t-win10-64-gpu-b gecko-t-win7-32-gpu-b'

AMI retention

OCC will automatically retain 10 previous images when building new images, so that rollback is possible via manual updating of the aws provisioner worker type definitions. However if you wish to rollback many versions, you may need to rebuild. In any case it is best to follow up a rollback with a git revert and push to master branch, so that the OCC repository is always in sync with the production versions.

OpenCloudConfig architecture

Here are some of the more important files in this repository.

File Description
/ci/update-workertype.sh CI script to rebuild AMIs and update live production worker type definitions.
/userdata/xDynamicConfig.ps1 Defines the allowed ComponentTypes accepted by OpenCloudConfig, as documented above.
/userdata/rundsc.ps1 Script to run on vanilla environment, to apply configuration.
/userdata/HaltOnIdle.ps1 Checks machine activity and decides whether to terminate it due to inactivity / bad state / etc.
/userdata/PrepLoaner.ps1 Prepares machine for a self-serve loan as per these instructions.
You can’t perform that action at this time.