feat: major overhaul to enable multiple builders to be supported in t…

…he all phases

* updates to provisioners
* scan adjustments
* refactored build, scanning and signing to remove Openstack dependencies
* removed requirement for a digest being provided as it'll now fetch it from the image metadata.
* added support for AMD gpus
* Added some new options for OpenStack

.github/workflows/pull_request.yaml

@@ -3,7 +3,7 @@ name: Validate on Pull Request
 on:
   workflow_dispatch:
   pull_request:
-    types: [ opened, reopened, synchronized ]
+    types: [ opened, reopened, synchronize ]
 
 jobs:
   lint:

.github/workflows/release.yml

@@ -32,7 +32,7 @@ jobs:
           CGO_ENABLED=0 go build -o "$BINARY_NAME" cmd/baski/main.go
       - name: Build Baski-Server
         run: |
-          CGO_ENABLED=0 go build -o "$SERVER_BINARY_NAME" cmd/baski/main.go
+          CGO_ENABLED=0 go build -o "$SERVER_BINARY_NAME" cmd/server/main.go
       - name: Release Notes
         run: |
           git log $(git describe HEAD~ --tags --abbrev=0)..HEAD --pretty='format:* %h %s%n  * %an <%ae>' --no-merges >> ".github/RELEASE-TEMPLATE.md"

CHANGELOG.md

@@ -1,9 +1,23 @@
 # Changelog
 
+## [ 2024/02/29 - v1.1.0 ]
+
+### BREAKING CHANGES
+
+* changed `cloud` prefix to `infra` in flags and config.
+* changed `build.nvidia` prefix to `build.gpu` in flags and config.
+* changed `build.nvidia.enable-nvidia-support` prefix to `build.gpu.enable-gpu-support` in flags and config.
+
+### Changed/Added
+
+* Added KubeVirt as a build option.
+* Supports AMD GPUs
+
 ## [ 2024/02/15 - v1.0.0 ]
 
 ### Changed/Added
+
 First release with:
 
 * Functioning support for OpenStack build, scan and signing.
-* Baski Server
+* Baski Server 

README.md

@@ -21,9 +21,10 @@ The scanning and signing functionality are separate from the build meaning these
 
 | Cloud Provider                 |
 |--------------------------------|
-| [Openstack](docs/openstack.md) |
+| [OpenStack](docs/openstack.md) |
+| [Kubevirt](docs/kubevirt.md)   |
 
-*More clouds could be supported but may not be maintained by Drewbernetes.*
+*More clouds could be supported but may not be maintained directly by Drewbernetes.*
 
 # Usage
 
@@ -47,7 +48,7 @@ For more flags and more info, run `baski --help`
 ### Running locally
 
 If you wish to run it locally then you can either build the binary and run it, or you can run it in docker by doing the
-following:
+following example for OpenStack:
 
 ```shell
 docker build -t baski:v0.0.0 -f docker/baski/Dockerfile .

baski-example.yaml

@@ -1,5 +1,8 @@
-# The cloud section defines the cloud to use.
-cloud:
+# The infra section defines the infrastructure to use.
+infra:
+  # type dictates which settings to use in the infra section below - must match the yaml key.
+  # IE, openstack, kubevirt etc
+  type: kubevirt
   # OpenStack support is a tricky one due to the varying nature of configurations that can exist across different setups.
   # The approach taken with this is a basic kolla-ansible install with no bells and whistles. Any additional configuration must be seriously considered before adding additional code/options to support it.
   openstack:
@@ -18,24 +21,53 @@ cloud:
     # Whether to use a floating IP on the instance.
     use-floating-ip: true
     # The name of the network from which to get a floating IP from.
-    floating-ip-network-name: "Internet"
+    floating-ip-network-name: "public1"
+    # Specify a specific security group to use instead of "default"
+    security-group: ""
     # Sets the image visibility once it has been created. The cloud account being used must have permission to do this.
     image-visibility: "public"
     # The disk format
     image-disk-format: "raw"
-    # the volume type
+    # Whether to use block_storage_volume
+    use-blockstorage-volume: false
+    # Specify an existing SSH keypair to use
+    ssh-keypair-name: "drew-rsa"
+    # The private Key file to use with the ssh-keypair-name
+    ssh-privatekey-file: "/home/<USER>/.ssh/id_rsa"
+    # The volume type
     volume-type: ""
     # The size of the storage volume
     volume-size: 0
     # The rootfs-UUID. In testing this has only been required for bare-metal instances as the bare-metal instance needs know about the rootfs, so it knows what to boot.
-    rootfs-uuid: "ROOT_FS_UUID"
+    rootfs-uuid: ""
+  # KubeVirt support is a wrapper around QEMU builds however at the end of the build it will generate a PVC in the target Kubernetes cluster
+  kubevirt:
+    # qemu_binary enables the overriding of the qemu-system-x86_64 binary name
+    qemu_binary: "qemu-system-x86_64"
+    # disk_size describes the size of the VM disk
+    disk_size: "10G"
+    # The directory in which the resulting QCOW2 image will be stored.
+    # This will have the image-name stored within it resulting in:
+    # /tmp/image-output/{{build.image-prefix}}-{{date}}-{{unique-code}}/{{build.build-os}}-kube-v{{build.kubernetes-version}}
+    output-directory: "/tmp/image-output"
+    # This enables the image to be uploaded to an S3 endpoint as defined in the S3 object and is recommended if using the datavolume-from-S3 approach in KubeVirt.
+    store-in-s3: false
+    # The bucket in which to store the resulting image - required if store-in-s3 is true
+    image-bucket: "images"
+    # kubevirt_namespace is the namespace in which the DV will be created
+    image-namespace: "baski-vm-images"
+# k8s contains kubernetes cluster options
+k8s:
+  # kubeconfig_path is the path to the kubeconfig that will be used to generate the PVC for Kubevirt
+  kubeconfig_path: "/tmp/kubeconfig"
 
-# Baski presumes there is an S3 endpoint available for the pulling of any items such as the .trivyignore and nvidia files.
-# This is because the image builder defaults to this for the Nvidia support, and it is presumed that most if not all people can have/can set up an S3 endpoint for authenticated storage.
+# Baski presumes there is an S3 endpoint available for the pulling of any items such as the .trivyignore and nvidia files (where required) as well as uploading any other items such as scan results and QEMU images (where required).
+# This is because the image builder defaults to this for the NVIDIA support, and it is presumed that most if not all people can have/can set up an S3 endpoint for authenticated storage.
 s3:
   endpoint: "S3_ENDPOINT_URL"
   access-key: "ACCESS_KEY"
   secret-key: "SECRET_KEY"
+  region: "us-east-1"
   # If the S3 endpoint is being used, set this to true to instruct the aws ansible role to do the same.
   is-ceph: true
 
@@ -46,10 +78,10 @@ build:
   # The OS to build for. This should match the source image.
   build-os: "ubuntu-2204"
   # The prefix to prepend to the name of the image that is built. The name will result in <prefix>-yymmdd-unique_id.
-  image-prefix: "kube"
+  image-prefix: "kmi"
   # The repo to use for image building. This will default to the main image builder repo but can be updated if additional functionality is required in a fork.
   image-repo: "https://github.com/kubernetes-sigs/image-builder.git"
-  # The branch to checkout from the image repo
+  # The branch to use in the image repo
   image-repo-branch: "main"
   # The crictl version.
   crictl-version: "1.26.0"
@@ -67,25 +99,40 @@ build:
   add-trivy: true
   # Whether to add Falco into the image.
   add-falco: true
-  # NVIDIA will soon be supported in the image builder https://github.com/kubernetes-sigs/image-builder/pull/1147.
-  # However, it has a prerequisite that means the operator should provide the NVIDIA license (.tok) and installer (.run) files via an S3 endpoint
-  # due to license restrictions by NVIDIA. These are not publicly available which is why this requirement is in place.
+  # GPU support is available in the image-builder project.
+  # NVIDIA support has a prerequisite that means the operator should provide the NVIDIA license (.tok) and installer (.run) files via an S3 endpoint
+  # due to license restrictions by NVIDIA. The vGPU drivers are not publicly available which is why this requirement is in place.
   # The image builder will not provide these files and if they are not supplied, the build will fail.
-  nvidia:
-    # enable NVIDIA driver install in the image.
-    enable-nvidia-support: true
-    # The NVIDIA driver version to be installed - Currently used for tagging metadata - may be removed in future in favor of parsing the filename.
-    nvidia-driver-version: "525.85.05"
-    # The S3 bucket to get the installer and license files from.
+  gpu:
+    # Enable GPU driver installs into the image.
+    enable-gpu-support: true
+    # The GPU architecture being used ("nvidia" or "amd" are currently supported).
+    gpu-vendor: "amd"
+    # The AMD driver version to be installed
+    amd-driver-version: "6.0.2"
+    # The AMD driver version to be installed
+    amd-deb-version: "6.0.60002-1"
+    # The AMDGPU Installer usecase
+    amd-usecase-version: "dkms"
+    # The NVIDIA driver version to be installed - Currently used for tagging metadata
+    nvidia-driver-version: "535.129.03"
+    # The S3 bucket to get the NVIDIA installer and license files from.
     nvidia-bucket: "nvidia"
-    # The installer file name in the bucket.
-    nvidia-installer-location: "NVIDIA-Linux-x86_64-525.85.05-grid.run"
-    # The license file name in the bucket.
+    # The NVIDIA installer file name in the bucket.
+    nvidia-installer-location: "NVIDIA-Linux-x86_64-535.129.03-grid.run"
+    # The NVIDIA license file name in the bucket.
     nvidia-tok-location: "client_configuration_token.tok"
-    # The feature type to configure the GRIDD service with -  see NVIDIA docs for more information on this.
+    # The feature type to configure the NVIDIA GRIDD service with - see NVIDIA docs for more information on this.
     nvidia-gridd-feature-type: "4"
   # The additional-images section should be a list of container images to bake into the image.
-  additional-images: [ ]
+  additional-images: []
+  # Any additional metadata/tags to add to the image as a map[string]string
+  # The following are available via the `generateBuilderMetadata` func in the provisioner and any additional data added here will be appended in that func
+  #  "os": o.BuildOS,
+  #  "k8s": o.KubeVersion,
+  #  "gpu": gpu,
+  #  "date": time.Now().Format(time.RFC3339),
+  additional-metadata: {}
 
 # Scan stage options
 scan:
@@ -96,7 +143,7 @@ scan:
   # Used in the `existing` command - takes a wildcard and scans all images that match it.
   multiple:
     # The image-search is used to filter images. If this string is within the name of the image, it'll be selected for scanning.
-    image-search: "kube-"
+    image-search: "kmi-"
     # How many concurrent scans to perform.
     concurrency: 2
   # Override the cloud.[provider].flavor for the scan. This can help avoid using a large or gpu enabled node just for scanning.
@@ -117,7 +164,7 @@ scan:
   trivyignore-filename: ".trivyignore"
   # A list of CVEs to add to the ignore list. If a file is also provided, this list will be appended to the list within the file.
   # If no file exists, then the file will be created with the list contents
-  trivyignore-list: [ ]
+  trivyignore-list: []
 
 # Sign stage options
 sign:
@@ -133,15 +180,12 @@ sign:
     # The Vault token.
     token: "VAULT_TOKEN"
     # The mount path within vault.
-    mount-path: "kv/baski"
-    # The name of the secret in the mount path.
+    mount-path: "baski"
+    # The name of the secret in the mount path - there should be a `password`, `private-key` and public-key` for cosign stored in here.
     secret-name: "signing-keys"
   # The ID of the image to sign.
   image-id: "" # Used for existing images
   # The private key to use in the signing process - this takes precedence over vault.
   private-key: ""
   # The public key to use in the validation process - this takes precedence over vault.
   public-key: ""
-  # If you would like to validate an image signing, this allows you to put the digest in and validate the image.
-  # This will be deprecated and then removed soon to make way for fetching the digest from the metadata of the image where it is currently stored.
-  digest: ""

docs/kubevirt.md

@@ -0,0 +1,13 @@
+### KubeVirt guidelines
+
+It is expected that you have a kubernetes cluster with KubeVirt installed already.
+
+If you wish to make use of the [CDI](https://kubevirt.io/user-guide/operations/containerized_data_importer/) support
+then it should be installed, and you should also have an S3 endpoint to which you can push the image that is built.
+
+Baski will push the image to the S3 endpoint then create a **DataVolume** (and required credentials if they don't exist)
+that will reference the image at the S3 endpoint as a source. 
+
+If you wish to manually do this yourself then set `kubevirt.store-in-s3` to false.
+
+Then craft a `baski.yaml` file based on the [example](../baski-example.yaml) supplied and run the commands you require.

docs/openstack.md

@@ -1,4 +1,4 @@
-### Openstack guidelines
+### OpenStack guidelines
 
 It is expected that you have a network and sufficient security groups in place to run this.<br>
 It will not create the network or security groups for you.