fix(deploy): use recreate strategy and 1 replica

[ebaron]

There are two bugs that occur due to replicas contending for access to the credentials database, and replicas on different nodes competing for the PVC itself. Cryostat is currently meant to work as a single replica, but during rolling updates there could be two simultaneously.

To avoid these issues, we're changing the deployment strategy from the default of RollingUpdate to Recreate. As a result there will be some brief downtime in response to upgrades or configuration changes. The other change is the operator managing the Replicas field of the main Cryostat deployment. This is to ensure that the replica count stays at 1 to avoid the above bugs.

To test:

1. make deploy create_cryostat_cr OPERATOR_IMG=....
2. kubectl edit cryostat cryostat-sample. Change some field such as setting reportOptions.replicas to 1.
3. The Cryostat deployment should scale down to 0 and then back up to 1, reflecting this new configuration.
4. Try manually scaling up/down the Cryostat deployment. After a brief time, the operator should revert the deployment to 1 replica.

Fixes: #487, #491

[tthvo]

kubectl edit cryostat cryostat-sample. Change some field such as setting reportOptions.replicas to 1.

About this, I am bit unsure what causes the main deployment to rollout new pods if this does not change its spec? Setting reportOptions.replicas just change the report deployment, right?

[tthvo]

I also got a bit of an issue when creating a report replica as pod seems to in CrashLoopBackOff with logs show:

exec /deployments/run-java.sh: permission denied

I am using Cluster Bot with OCP 4.11.1

[ebaron]

kubectl edit cryostat cryostat-sample. Change some field such as setting reportOptions.replicas to 1.

About this, I am bit unsure what causes the main deployment to rollout new pods if this does not change its spec? Setting reportOptions.replicas just change the report deployment, right?

This ends up setting the CRYOSTAT_REPORT_GENERATOR environment variable with the URL to the reports service:

cryostat-operator/internal/controllers/common/resource_definitions/resource_definitions.go

Lines 661 to 668 in 23cfadd

	if specs.ReportsURL != nil {
	reportsEnvs := []corev1.EnvVar{
	{
	Name: "CRYOSTAT_REPORT_GENERATOR",
	Value: specs.ReportsURL.String(),
	},
	}
	envs = append(envs, reportsEnvs...)

.

So there is a subtle config change on the main deployment as well.

[ebaron]

I also got a bit of an issue when creating a report replica as pod seems to in CrashLoopBackOff with logs show:

exec /deployments/run-java.sh: permission denied

I am using Cluster Bot with OCP 4.11.1

Oh this again. We've encountered this with some of our sample apps. It's a matter of updating our Dockerfiles. You should still be able to test this PR though. Instead of the reports replicas, you could try switching enableCertManager.

[tthvo]

kubectl edit cryostat cryostat-sample. Change some field such as setting reportOptions.replicas to 1.

About this, I am bit unsure what causes the main deployment to rollout new pods if this does not change its spec? Setting reportOptions.replicas just change the report deployment, right?

This ends up setting the CRYOSTAT_REPORT_GENERATOR environment variable with the URL to the reports service:

cryostat-operator/internal/controllers/common/resource_definitions/resource_definitions.go

Lines 661 to 668 in 23cfadd

	if specs.ReportsURL != nil {
	reportsEnvs := []corev1.EnvVar{
	{
	Name: "CRYOSTAT_REPORT_GENERATOR",
	Value: specs.ReportsURL.String(),
	},
	}
	envs = append(envs, reportsEnvs...)

.

So there is a subtle config change on the main deployment as well.

Ah niceee! I didnt notice before. Make sense! Thanks :D

[tthvo]

Oh this again. We've encountered this with some of our sample apps. It's a matter of updating our Dockerfiles. You should still be able to test this PR though. Instead of the reports replicas, you could try switching enableCertManager.

Nice, so its unrelated then. And looks great to me and deployments are recreated as expected. And thanks a lot for explaining those above :D