Add a new resource to manipulate AWS AMI

[olblak]

Add Amazon AMI resource

• Documentation

This pull request introduces to retrieve AMi using a source resource or to test if a AMI exist.

Fix #222

Test

To test this pull request, you can run the following commands:

  cp pkg/plugins/aws/ami/
  go test

! Required authentication

Additionnal Information

Settings

Available updatecli configuration
...

Authentication

Different mechanisms can be used to authenticate with AWS API.

Environment Variables

updatecli retrieves credentials from the environment variables. Environment credentials never expire.

Environment variables used:

• Access Key ID: AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY

• Secret Access Key: AWS_SECRET_ACCESS_KEY or AWS_SECRET_KEY

Shared Credentials File

By default updatecli looks for "AWS_SHARED_CREDENTIALS_FILE" env variable. If the
env value is empty will default to the current user's home directory.
Linux/OSX: "$HOME/.aws/credentials"
Windows: "%USERPROFILE%.aws\credentials"

AWS Profile to extract credentials from the shared credentials file. If empty
will default to environment variable "AWS_PROFILE" or "default" if
the environment variable is also not set.

Updatecli configuration

We can provide credentials directly from a updatecli configuration by using the two following settings

.updatecli configuration

kind: aws/ami
name: resource to manipulate aws ami id
spec:
  access_key: "insert aws access key"
  secret_key: "insert aws secret key"
  filters:
    - name: "name"
      values: "image name "

Filters

Different filters can be used by updatecli to know which ami-id to look after.

.updatecli configuration

kind: aws/ami
name: resource to manipulate aws ami id
spec:
  access_key: "insert aws access key"
  secret_key: "insert aws secret key"
  filters:
    - name: "name"
      values: "image name "

The filters.

• architecture - The image architecture (i386 | x86_64 | arm64).

• block-device-mapping.delete-on-termination - A Boolean value that indicates
  whether the Amazon EBS volume is deleted on instance termination.

• block-device-mapping.device-name - The device name specified in the
  block device mapping (for example, /dev/sdh or xvdh).

• block-device-mapping.snapshot-id - The ID of the snapshot used for the
  EBS volume.

• block-device-mapping.volume-size - The volume size of the EBS volume,
  in GiB.

• block-device-mapping.volume-type - The volume type of the EBS volume
  (gp2 | io1 | io2 | st1 | sc1 | standard).

• block-device-mapping.encrypted - A Boolean that indicates whether the
  EBS volume is encrypted.

• description - The description of the image (provided during image creation).

• ena-support - A Boolean that indicates whether enhanced networking with
  ENA is enabled.

• hypervisor - The hypervisor type (ovm | xen).

• image-id - The ID of the image.

• image-type - The image type (machine | kernel | ramdisk).

• is-public - A Boolean that indicates whether the image is public.

• kernel-id - The kernel ID.

• manifest-location - The location of the image manifest.

• name - The name of the AMI (provided during image creation).

• owner-alias - The owner alias (amazon | aws-marketplace). The valid
  aliases are defined in an Amazon-maintained list. This is not the AWS
  account alias that can be set using the IAM console. We recommend that
  you use the Owner request parameter instead of this filter.

• owner-id - The AWS account ID of the owner. We recommend that you use
  the Owner request parameter instead of this filter.

• platform - The platform. To only list Windows-based AMIs, use windows.

• product-code - The product code.

• product-code.type - The type of the product code (devpay | marketplace).

• ramdisk-id - The RAM disk ID.

• root-device-name - The device name of the root device volume (for example,
  /dev/sda1).

• root-device-type - The type of the root device volume (ebs | instance-store).

• state - The state of the image (available | pending | failed).

• state-reason-code - The reason code for the state change.

• state-reason-message - The message for the state change.

• sriov-net-support - A value of simple indicates that enhanced networking
  with the Intel 82599 VF interface is enabled.

• tag: - The key/value combination of a tag assigned to the resource.
  Use the tag key in the filter name and the tag value as the filter value.
  For example, to find all resources that have a tag with the key Owner
  and the value TeamA, specify tag:Owner for the filter name and TeamA for
  the filter value.

• tag-key - The key of a tag assigned to the resource. Use this filter
  to find all resources assigned a tag with a specific key, regardless of
  the tag value.

• virtualization-type - The virtualization type (paravirtual | hvm).

Tradeoff

Potential improvement

[dduportal]

Played with the following example:

---
title: Bump agent templates version on all controllers
sources:
  packerImageVersion:
    kind: githubRelease
    spec:
      owner: "jenkins-infra"
      repository: "packer-images"
      token: "{{ requiredEnv .github.token }}"
      username: "{{ .github.username }}"
  getLatestUbuntuAgentAMIAmd64:
    kind: aws/ami
    depends_on:
      - packerImageVersion
    spec:
      region: us-east-2
      filters:
        - name: "name"
          values: "jenkins-agent-ubuntu-20-amd64-*"
        - name: "tag:build_type"
          values: "prod"
        - name: "tag:version"
          values: '{{ source "packerImageVersion" }}'
  getLatestWindowsAgentAMIAmd64:
    kind: aws/ami
    depends_on:
      - packerImageVersion
    spec:
      region: us-east-2
      filters:
        - name: "name"
          values: "jenkins-agent-windows-2019-amd64-*"
        - name: "tag:build_type"
          values: "prod"
        - name: "tag:version"
          values: '{{ source "packerImageVersion" }}'
  getLatestUbuntuAgentAMIArm64:
    kind: aws/ami
    depends_on:
      - packerImageVersion
    spec:
      region: us-east-2
      filters:
        - name: "name"
          values: "jenkins-agent-ubuntu-20-arm64-*"
        - name: "tag:build_type"
          values: "prod"
        - name: "tag:version"
          values: '{{ source "packerImageVersion" }}'

targets:
  setUbuntuAgentAMIAmd64:
    name: "Bump AMI ID for Ubuntu AMD64 agents"
    kind: shell
    sourceID: getLatestUbuntuAgentAMIAmd64
    spec:
      command: echo ubuntu-amd
  setUbuntuAgentAMIArm64:
    name: "Bump AMI ID for Ubuntu ARM64 agents"
    kind: shell
    sourceID: getLatestUbuntuAgentAMIArm64
    spec:
      command: echo ubuntu-arm
  setWindowsAgentAMIAmd64:
    name: "Bump AMI ID for Windows ARM64 agents"
    kind: shell
    sourceID: getLatestWindowsAgentAMIAmd64
    spec:
      command: echo windows-amd

• First of all it works flawlessly 👏 🥳 it's really fast, and used my (weird) AWS config
• The only issue I got is how to handle when there are multiple results to the query.

On my current use case, i'm retrieving the latest image by sorting on the CreationDate attribute, using the --query flag of the aws CLI:

aws ec2 describe-images \
  --owners 200564066411 `# Owner ID for AWS account cloudbees-jenkins` \
  --region=us-east-2 `#AWS region where ci.jenkins.io runs its EC2 agents` \
  --filters "Name=name,Values=jenkins-agent-${operating_system}-${cpu_architecture}-*" `# Search by name, with a pattern for the end (timestamps)` \
    "Name=tag:build_type,Values=prod" `# Only retrieve production ready AMIs` \
    "Name=tag:version,Values=${image_version}" `# image version is a semantic version v2 string` \
  --query 'sort_by(Images, &CreationDate)[-1].[ImageId]' `# Get the most recent one by sorting by creation date and get the latest and extract its AMI ID` \
  --output text `# We don't want JSON`

[olblak]

@dduportal I added a new spec key "sortBy: creationDateAsc" that you can use to order the list of AMI ID, the resource still returns the latest AMI found but we can now decide to sort the result by ascending or descending based on creation date.

Sortby can now accept the following values ["","creationDateAsc", "creationDateDesc"] with default set to ""
So

    getLatestWindowsAgentAMIAmd64:
      kind: aws/ami
      spec:
        region: us-east-2
        sortBy: creationDateAsc
        filters:
          - name: "name"
            values: "jenkins-agent-windows-2019-amd64-*"
          - name: "tag:build_type"
            values: "prod"
          - name: "tag:version"
            values: '0.5.0'

is equivalent to

aws ec2 describe-images --owners 200564066411 --region=us-east-2 --filters 'Name=name,Values=jenkins-agent-windows-2019-amd64-*' Name=tag:build_type,Values=prod Name=tag:version,Values=0.5.0 --query 'sort_by(Images, &CreationDate)[-1].ImageId'

[dduportal]

@olblak that sounds really cool! thanks for the extra effort!

[dduportal]

🚀 Works really well!