Break the current operator's configuration into custom and managed properties

[dmartinol]

Closes #381

• Added operator managed ConfigMap to store the managed properties (aka immutable)
  • Key application-PROFILE.properties contains the properties (PROFILE is either prod or dev)
• User properties are no more updated by the operator
  • Properties are defined by the key application.properties
• Any changes to one of the two property files implies a forced rollout of the workflow deployment
  • The checksum is calculated on the joined properties (user + managed)
• Since the managed properties are generated using the SonataFlowPlatform instance, all the creator and mutators have been extended to add a platform instance to the input parameters

More tests can be added after we agree on the overall approach.

[ricardozanini]

We can add this link to the documentation: https://quarkus.io/guides/config-reference#default-profiles

[ricardozanini]

@dmartinol can you please verify the e2e failures?

Summarizing 1 Failure:
  [FAIL] SonataFlow Operator ensure that Operator and Operand(s) can run in restricted namespaces [It] should successfully deploy the orderprocessing workflow in devmode and verify if it's running
  /home/runner/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.13.0/internal/node.go:463

Might be related with these changes.

[ricardozanini]

I'll take a closer look later, but looks ok at a first glance. I let a proposal to avoid including the platform in every creator/mutator.

[dmartinol]

@dmartinol can you please verify the e2e failures?

Summarizing 1 Failure:
  [FAIL] SonataFlow Operator ensure that Operator and Operand(s) can run in restricted namespaces [It] should successfully deploy the orderprocessing workflow in devmode and verify if it's running
  /home/runner/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.13.0/internal/node.go:463

Might be related with these changes.

I see the managed properties do not contain the "converted" discovery properties, so there was not mapping for this configuration:

mp.messaging.outgoing.kogito_outgoing_stream.url = ${kubernetes:services.v1/event-listener}

I'm investigating

[dmartinol]

We can add this link to the documentation: https://quarkus.io/guides/config-reference#default-profiles

can you remind me again the documentation repo?

[dmartinol]

@ricardozanini apart from some E2E tests that are failing and I cannot replicate on my local environment, unit tests and manual verification seems to succeed. I will update the ADR in the meantime

[ricardozanini]

We can add this link to the documentation: https://quarkus.io/guides/config-reference#default-profiles

can you remind me again the documentation repo?

Sure thing! https://github.com/apache/incubator-kie-kogito-docs/tree/main/serverlessworkflow/modules/ROOT

[dmartinol]

We can add this link to the documentation: https://quarkus.io/guides/config-reference#default-profiles

Adding to PR for doc repo

[ricardozanini]

Looks neat! I just have two questions.

[ricardozanini]

I think you can rebase this PR to take latest Jordi changes to the tests.

[dmartinol]

I'm touched by that green flag on the E2E checks 😂

[wmedvede]

@dmartinol looks like we need some love here to fix merge conflicts

[dmartinol]

@dmartinol looks like we need some love here to fix merge conflicts

and also updated some props, but I need to re-run my manual tests, I'm concerned that latest commits broke anything.
BTW: about properties-related tests, it doesn't make much sense that we only check the number of generated props, but at this point I cannot add a check for each expected properties, as this set is changing day by day. So, I just changed the expected number.

[ricardozanini]

@dmartinol I agree. Can you open a follow up issue? So I can track it.

[wmedvede]

I have left some minor comments, good job!

[wmedvede]

@dmartinol @ricardozanini @domhanak some positive feedback for the record.

I have executed usecase-oc https://github.com/flows-examples/techpreview2/blob/f39a25e626e217dabf7b69237641ca1e05375c6a/README.md?plain=1#L739

And I can see this results:

Managed and normal properties are created

I've added a service discovery property manually in the callbackstate-props

Then I went to the managed properties and I can see that the service discovery was produced and the corresponding property is in the managed set. The other managed properties are still there. good!

• The workflow is running well I could create instances and see them in the DI, good!

• When I add properties manually to the user properties I can then see the workflow rollingout. good!

• In the workflow deployment environment I can't see any property
  But maybe we can start moving things there in a follow-up PR. If I understood well, we should start to see there this kind of props
  quarkus.http.port = 8080
  quarkus.http.host = 0.0.0.0
  as env vars.

But we can think about this in follow-up PR

good job @dmartinol !!!!

[wmedvede]

@dmartinol @ricardozanini approving from my side after doing some local testing since my comments are minor suggestions.

[dmartinol]

@dmartinol @ricardozanini @domhanak some positive feedback for the record.

• In the workflow deployment environment I can't see any property
  But maybe we can start moving things there in a follow-up PR. If I understood well, we should start to see there this kind of props
  quarkus.http.port = 8080
  quarkus.http.host = 0.0.0.0
  as env vars.

But we can think about this in follow-up PR

Thanks for deep investigation and testing.
Wrt the deployment environment, we changed our initial design because this option doesn't work well with property names including "-", see this issue smallrye-reactive-messaging: incompatibility with configuration through environment variables. E.g., mp.messaging.outgoing.kogito-job-service-job-request-events.connector cannot be managed that way.

If you aim to model as env variables only the properties not affected by that issue, this is an option but at the same time we should keep the managed properties file for all of the others, so we'd have 3 managed source of configuration values.

[wmedvede]

@dmartinol @ricardozanini @domhanak

I have found a small non blocking issue here.

When playing to add service discovery properties in the callbackstatetimeouts-prop I define this:

After the managed properties calculation was produced in the Reconcile, I can see that the service discovery properties where created, good, but, user.property3 and user.property4 was copied to the managed properties too with a sort of calculated value.
This didn't happen before. It looks like a property expansion is enabled now during the properties calculation or something similar.

[wmedvede]

@dmartinol @ricardozanini @domhanak some positive feedback for the record.

• In the workflow deployment environment I can't see any property
  But maybe we can start moving things there in a follow-up PR. If I understood well, we should start to see there this kind of props
  quarkus.http.port = 8080
  quarkus.http.host = 0.0.0.0
  as env vars.

But we can think about this in follow-up PR

Thanks for deep investigation and testing. Wrt the deployment environment, we changed our initial design because this option doesn't work well with property names including "-", see this issue smallrye-reactive-messaging: incompatibility with configuration through environment variables. E.g., mp.messaging.outgoing.kogito-job-service-job-request-events.connector cannot be managed that way.

If you aim to model as env variables only the properties not affected by that issue, this is an option but at the same time we should keep the managed properties file for all of the others, so we'd have 3 managed source of configuration values.

ok, we can see this in a future step.

[ricardozanini]

@wmedvede I'd rather keep all managed properties in the CM. One place to look for the managed properties by the operator can facilitate admin tasks.

Regarding the bug you've found, this is intentional to override the ones by the user. Otherwise, Quarkus will try to read them and parse. Since kubernetes: is not a valid schema, it will fail.

Actually, by chance, @dmartinol found a way to remove our microprofile-config add-on. We don't need it anymore since it's already doing the expansion on the operator side. So in your example, only these will do:

application-dev.properties (Managed)

user.property3=<discovered URL>
user.property4=<discovered URL>

application.properties (User)

user.property3=<desired Kubernetes endpoint>
user.property4=<desired Kubernetes endpoint>

We can ditch the jar and live happily.

[wmedvede]

We can ditch the jar and live happily.

@ricardozanini
np from my side, the only thing is that I personally don't know, is how these properties has ended up in the managed property set, and also with the calculated values

user.property3=
user.property4=

Then, regarding the microprofile like generated properties and the removal of that addon, I think we need to investigate a bit, because when we configure knative-serving operations like:

  "operation": "knative:services.v1.serving.knative.dev/default/serverless-workflow-greeting-quarkus?path=/arrayFunction"

To make them work, if I'm not wrong we need the microprofile addon, need to double check on monday.

[ricardozanini]

@dmartinol can you please try rebasing? Since we've merged the label PR, I think this line in the e2e is causing troubles:

kubectl wait pod -A -l control-plane=sonataflow-operator --for condition=Ready

Your fork still has the old label: https://github.com/dmartinol/incubator-kie-kogito-serverless-operator/blob/SRVLOGIC-195-props/.github/workflows/e2e.yml#L74

[ricardozanini]

need to double check on Monday.

@wmedvede no worries. The problem is that our addon should consider the property in the default profile, which is not. That's why we had to override in the managed CM.

[ricardozanini]

@dmartinol once you rebase, CI is green, I'll merge this.