[release-v1.59] Forklift controller (#3261)

* Run bazelisk run //robots/cmd/uploader:uploader -- -workspace /home/prow/go/src/github.com/kubevirt/project-infra/../containerized-data-importer/WORKSPACE -dry-run=false (#3247)

Signed-off-by: kubevirt-bot <kubevirtbot@redhat.com>

* Improve usability of Golangci-lint (#3193)

* Run 'make format && make generate'

Seems like this hadn't been run in a long while.

Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>

* Run 'golangci-lint --fix' when using 'make format'

Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>

* Update golangci-lint version

Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>

* Remove gofmt

Golangci-lint already runs gofmt internally (see ./.golangci.yml)

Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>

* Remove golint pass

This linter has been deprecated since May 9, 2021. Golangci-lint already
covers its functionality

Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>

* Remove goimports

Golangci-lint already runs goimports internally (see ./.golangci.yml)

Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>

* Move govet into golangci-lint

This way we don't parse the project twice.

Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>

* Update README

Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>

* Run formaters and generators again

Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>

---------

Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>

* forklift: introduce forklift controller (#2983)

* forklift: add types for forklift populators

Introduce the OvirtVolumePopulator and OpenstackVolumePopulator types

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

* forklift: introduce forklift controller

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

* forklift: add CRD missing CRD suffix

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

* forklift: fix tests

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

* forklift: start support for mutated PVC

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

* forklift: fix linter issues

and skip bound PVCs

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

* forklift: update vendor and generated code

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

* forklift: remove unnecessary argument

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

* forklift: add ovirt-populator to cdi-importer

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

* forklift: fix linter issues

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

* forklift: fix contrller_test

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

* forklift: regenerate swagger

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

* forklift: normalize ovirt image name

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

* forklift: restore OVIRT_POPULATOR_IMAGE_NAME

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

* forklift: address comments

- remove accidental copy-paste
- change type of NAD reference to string
- use NAD in the controller

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

* forklift: add doc with examples for the forklift populators

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

* forklift: rename secret reference

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

* forklift: fix linter warning

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

* forklift: update deps

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

* forklift: fix linter issues

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

* forklift: use ginkgo for openstack populator test

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

* Roll back nginx from 1.24.0 to 1.22.1 to avoid segfaults when pulling…
… images with tls

Possibly a bug, there were some segfault fixes in 1.25.4 release:
https://nginx.org/en/CHANGES

Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>

* forklift: address comments

- Fix const
- Reverse conditional
- Pass context to reconcile
- Improve cross namespace check
- Add populator pod watcher

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

* forklift: address comments

- Drop unnecessary comments
- Add dataSourceRef check
- Add node to spec

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

* forklift: add restartCount annotation

- fix race condition leaving Running phase in annotation
- fix metrics UT

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

* forklift: address comments

- Change optional fields to a pointer
- Fix Pod watch
- Add NotFound check to avoid an extra reconile

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

* forklift: regenerate

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>

---------

Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
Co-authored-by: Alex Kalenyuk <akalenyu@redhat.com>

* Run bazelisk run //robots/cmd/uploader:uploader -- -workspace /home/prow/go/src/github.com/kubevirt/project-infra/../containerized-data-importer/WORKSPACE -dry-run=false

Signed-off-by: kubevirt-bot <kubevirtbot@redhat.com>

---------

Signed-off-by: kubevirt-bot <kubevirtbot@redhat.com>
Signed-off-by: Edu Gómez Escandell <egomez@redhat.com>
Signed-off-by: Benny Zlotnik <bzlotnik@redhat.com>
Signed-off-by: Alex Kalenyuk <akalenyu@redhat.com>
Co-authored-by: kubevirt-bot <kubevirtbot@redhat.com>
Co-authored-by: Edu Gómez Escandell <egomez@redhat.com>
Co-authored-by: Benny Zlotnik <2139890+bennyz@users.noreply.github.com>

.golangci.yml

@@ -8,6 +8,7 @@ linters:
   - ginkgolinter
   - gofmt
   - goimports
+  - govet
   - misspell
   - nakedret
   - unconvert

cmd/cdi-controller/BUILD.bazel

@@ -17,6 +17,7 @@ go_library(
         "//pkg/util/cert/fetcher:go_default_library",
         "//pkg/util/cert/generator:go_default_library",
         "//staging/src/kubevirt.io/containerized-data-importer-api/pkg/apis/core/v1beta1:go_default_library",
+        "//staging/src/kubevirt.io/containerized-data-importer-api/pkg/apis/forklift/v1beta1:go_default_library",
         "//vendor/github.com/kelseyhightower/envconfig:go_default_library",
         "//vendor/github.com/kubernetes-csi/external-snapshotter/client/v6/apis/volumesnapshot/v1:go_default_library",
         "//vendor/github.com/openshift/api/config/v1:go_default_library",

cmd/cdi-controller/controller.go

@@ -37,6 +37,7 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/manager/signals"
 
 	cdiv1 "kubevirt.io/containerized-data-importer-api/pkg/apis/core/v1beta1"
+	forklift "kubevirt.io/containerized-data-importer-api/pkg/apis/forklift/v1beta1"
 	"kubevirt.io/containerized-data-importer/pkg/common"
 	"kubevirt.io/containerized-data-importer/pkg/controller"
 	dvc "kubevirt.io/containerized-data-importer/pkg/controller/datavolume"
@@ -60,6 +61,7 @@ var (
 	clonerImage            string
 	uploadServerImage      string
 	uploadProxyServiceName string
+	ovirtPopulatorImage    string
 	configName             string
 	pullPolicy             string
 	verbose                string
@@ -71,6 +73,7 @@ var (
 		cdiv1.AddToScheme,
 		extv1.AddToScheme,
 		snapshotv1.AddToScheme,
+		forklift.AddToScheme,
 		imagev1.Install,
 		ocpconfigv1.Install,
 		routev1.Install,
@@ -103,6 +106,7 @@ func init() {
 	importerImage = getRequiredEnvVar("IMPORTER_IMAGE")
 	clonerImage = getRequiredEnvVar("CLONER_IMAGE")
 	uploadServerImage = getRequiredEnvVar("UPLOADSERVER_IMAGE")
+	ovirtPopulatorImage = getRequiredEnvVar("OVIRT_POPULATOR_IMAGE")
 	uploadProxyServiceName = getRequiredEnvVar("UPLOADPROXY_SERVICE")
 	installerLabels = map[string]string{}
 
@@ -301,6 +305,10 @@ func start() {
 		klog.Errorf("Unable to setup clone populator: %v", err)
 		os.Exit(1)
 	}
+	if _, err := populators.NewForkliftPopulator(ctx, mgr, log, importerImage, ovirtPopulatorImage, installerLabels); err != nil {
+		klog.Errorf("Unable to setup forklift populator: %v", err)
+		os.Exit(1)
+	}
 
 	klog.V(1).Infoln("created cdi controllers")
 

cmd/cdi-importer/BUILD.bazel

@@ -46,6 +46,7 @@ container_image(
     files = [
         ":cdi-importer",
         "//cmd/openstack-populator",
+        "//cmd/ovirt-populator",
         "//tools/cdi-containerimage-server",
         "//tools/cdi-image-size-detection",
         "//tools/cdi-source-update-poller",

cmd/openstack-populator/BUILD.bazel

@@ -1,17 +1,18 @@
-load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library")
+load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library", "go_test")
 
 go_library(
     name = "go_default_library",
     srcs = ["openstack-populator.go"],
     importpath = "kubevirt.io/containerized-data-importer/cmd/openstack-populator",
     visibility = ["//visibility:private"],
     deps = [
+        "//pkg/util/prometheus:go_default_library",
         "//vendor/github.com/gophercloud/gophercloud:go_default_library",
         "//vendor/github.com/gophercloud/gophercloud/openstack:go_default_library",
         "//vendor/github.com/gophercloud/gophercloud/openstack/imageservice/v2/imagedata:go_default_library",
         "//vendor/github.com/gophercloud/utils/openstack/clientconfig:go_default_library",
         "//vendor/github.com/prometheus/client_golang/prometheus:go_default_library",
-        "//vendor/github.com/prometheus/client_golang/prometheus/promhttp:go_default_library",
+        "//vendor/github.com/prometheus/client_model/go:go_default_library",
         "//vendor/k8s.io/klog/v2:go_default_library",
     ],
 )
@@ -21,3 +22,16 @@ go_binary(
     embed = [":go_default_library"],
     visibility = ["//visibility:public"],
 )
+
+go_test(
+    name = "go_default_test",
+    srcs = [
+        "openstack-populator_test.go",
+        "openstack_populator_suite_test.go",
+    ],
+    embed = [":go_default_library"],
+    deps = [
+        "//vendor/github.com/onsi/ginkgo/v2:go_default_library",
+        "//vendor/github.com/onsi/gomega:go_default_library",
+    ],
+)