CSPL-3064: Support for Distroless Image Creation in Splunk Operator for Kubernetes

[vivekr-splunk]

Summary

This PR adds support for creating a distroless image for the Splunk Operator for Kubernetes alongside the existing UBI (Universal Base Image) image. Instead of replacing the UBI image, we will provide both UBI and distroless images to allow flexibility for different customer requirements. This enhancement improves security by reducing the attack surface of the operator container while maintaining compatibility with existing workflows.

---

Why This Change Is Needed

Currently, the Splunk Operator uses the UBI image, which includes a complete Linux environment with various tools and libraries that may not be necessary for running the operator. By supporting a distroless image, we can:

1. Improve Security: Distroless images do not contain shells, package managers, or other unnecessary utilities, reducing potential vulnerabilities.
2. Reduce Attack Surface: Fewer tools in the image mean fewer opportunities for attackers to exploit.
3. Decrease Image Size: Distroless images are smaller, leading to faster deployments and lower bandwidth usage.

---

Changes Implemented

1. Makefile Update:

   • Added logic to detect whether a distroless base image is being used.
   • Modified the docker-buildx target to support building both UBI and distroless images.
   • The distroless image is tagged with a -distroless suffix (<image-name>-distroless).

2. Dockerfile Update:

   • Modified the existing Dockerfile to support building a distroless image.
   • Introduced a new Dockerfile.distroless with only essential dependencies required for runtime.

3. CI/CD Pipeline:

   • Updated the CI/CD pipeline to build and push both UBI and distroless images.
   • Ensured that integration and smoke tests are run against both images to validate functionality.

4. Documentation:

   • Updated the README and user guides to document the distroless image option.
   • Provided guidelines for customers on how to choose between the UBI and distroless images based on their use cases.

---

Benefits

• Improved Security: Distroless images lack a shell, package manager, or other utilities, making it harder for attackers to exploit the container.
• Reduced Attack Surface: By minimizing the container content to only essential components, we reduce potential entry points for attackers.
• Faster Deployments: The smaller size of the distroless image results in faster deployments and reduced bandwidth usage.

---

Impact

1. Backward Compatibility: The UBI image will continue to be supported, ensuring backward compatibility with existing deployments.
2. Customer Debugging: Since distroless images lack a shell, debugging can be challenging. To address this, we propose implementing a sidecar container or an alternative debugging mechanism to allow customers to inspect volumes and files.
3. Testing: This change requires thorough testing of both images to ensure that the operator continues to function correctly across all scenarios.

---

Tasks Completed

• Modified Dockerfile and created a new Dockerfile.distroless.
• Updated the Makefile to build both UBI and distroless images.
• Validated the operator functionality with distroless images through integration and smoke tests.
• Updated CI/CD pipelines to build and push both images.
• Updated documentation and user guides to reflect the changes.

---

Next Steps

• Implement and test a sidecar container or an alternative approach for customer debugging.
• Document the debugging mechanism and update user guides accordingly.

---

How to Test

1. Build the UBI Image:

   make docker-buildx IMG=<image-name> BASE_IMAGE=registry.access.redhat.com/ubi8/ubi

2. Build the Distroless Image:

   make docker-buildx IMG=<image-name> BASE_IMAGE=gcr.io/distroless/static:nonroot

3. Run Integration Tests:

   make int-test

---

Priority: High

This change enhances the security posture of the Splunk Operator and aligns with best practices for container image minimalism.

---

Reviewer Notes:
Please review the changes to the Dockerfile, Makefile, and CI/CD pipeline. Special attention should be given to the sidecar proposal for customer debugging.

[patrykw-splunk]

LGTM

[Igor-splunk]

Not sure about the CR tag name in merge-develop-to-main workflow, distroless should probably be at the end of the tag.

[akondur]

Should we add a parameter here to pass the dockerfile used to build? If we change the file names in the future, it might be easier to incorporate the change.

[Igor-splunk]

Tests actually pased here -> https://github.com/splunk/splunk-operator/actions/runs/12999543813/job/36255541155
for commit c548bb8