AbstractDependentResource should be able to pass his turn

[scrocquesel]

Actually, when the secret is not yet created, the mysql schema sample is throwing some errors in SchemaDependentResource due to orElseThrow.

https://github.com/java-operator-sdk/java-operator-sdk/blob/2eac098db872cedcefecb6036ec98c77b58cf60f/sample-operators/mysql-schema/src/main/java/io/javaoperatorsdk/operator/sample/dependent/SchemaDependentResource.java#L56

The same apply with the reconciler

https://github.com/java-operator-sdk/java-operator-sdk/blob/2eac098db872cedcefecb6036ec98c77b58cf60f/sample-operators/mysql-schema/src/main/java/io/javaoperatorsdk/operator/sample/MySQLSchemaReconciler.java#L39

I'm trying to avoid these exceptions because there are not really exceptions, but just the fact that some requirement are not yet satisfied. So I tried,

@Override
    public Schema desired(MySQLSchema primary, Context<MySQLSchema> context) {
        return context.getSecondaryResource(Secret.class)
                .map(secret -> new Schema(primary.getMetadata().getName(), primary.getSpec().getEncoding()))
                .orElse(null);
    }

But AbstractDependentResource.logForOperation produces an NPE and even if it doesn't, I think that create should not be call.

It is not clear what to do in such a case. I don't feel like exception are the way to go because it raise a call to updateErrorStatus when it should definitely not. Should desired return an Optional<R> ?

[metacosm]

I wanted to have an ERROR case in ReconcileResult but we decided that this was too much of a behavioral change at this point. That said, I agree with you, right now, we're using exceptions for behavior / control and it's a bad design, imo.

[csviri]

How can it happen that the secret is not created (I mean other than an error state?). The secret is a managed dependent resource that is created before the schema. Managed dependent resources are created in order.

It's true that lately the behavior changed, that we reconcile all the resources event if the previous ones failed. This makes issues when the first operation fails, but only if the resources are not independent.

In this case a fail fast approach would be better in the current scope. So not reconcile resources if one fails before.

[csviri]

I wanted to have an ERROR case in ReconcileResult but we decided that this was too much of a behavioral change at this point. That said, I agree with you, right now, we're using exceptions for behavior / control and it's a bad design, imo.

The ERROR does not fit in the big picture now, could you create a separate issue for that to discuss the details?

[metacosm]

This is the issue, right here, I think :)

[csviri]

I'm trying to avoid these exceptions because there are not really exceptions, but just the fact that some requirement are not yet satisfied. So I tried,

Again, what are the reason that are not satisfied yet in your case? @scrocquesel
So why is the secret not created.

[csviri]

This is the issue, right here, I think :)

Maybe with a nice description, definition of the problem in general..

[csviri]

TBH I would revert this into a fail fast behavior, that would solve this problem, and would be much more easier to follow what happening from the logs.

[metacosm]

This is the issue, right here, I think :)

Maybe with a nice description, definition of the problem in general..

The problem is as explained in the description of the issue: people want to be able to make decisions based on the reconcile result of dependents and right now, the reconcile result doesn't cover all the possibilities. This should be part of a more general approach, of course, but that's the core of the issue. It might just be that the dependent is not yet ready, for example. It shouldn't be an error that triggers a retry because this might cause retries to get exhausted before your dependent ever gets ready, so managing this situation via exceptions is not correct, imo.

[csviri]

The problem is as explained in the description of the issue: people want to be able to make decisions based on the reconcile result of dependents and right now, the reconcile result doesn't cover all the possibilities. This should be part of a more general approach, of course, but that's the core of the issue. It might just be that the dependent is not yet ready, for example. It shouldn't be an error that triggers a retry because this might cause retries to get exhausted before your dependent ever gets ready, so managing this situation via exceptions is not correct, imo.

1. That is a completely different/additional feature. Like waiting for something to be ready is completely out of scope, ( should be managed inside the reconcile() if that needs to be synchronouse?). This now can be handled with standalone mode btw (cc @scrocquesel ). But Secret has no concept of ready anyways. Also we agreed on a different thread that waiting synchronously even for a Deployment does not make sense in general.

2. What is in this issue is basically error handling. But as we discussed before, this will be part of a bigger picture, declaratively describe workflows. Ideally imo this should be discussed also taking in mind that. see: Workflows for Dependent Resource - Umbrella Issue #1097

3. Handling an explicit error state would not be "seamless" integration into the current exception handling model. What is based on fail fast approach normally a generic error handler (analogy to try-catch block), see ErrorStatusHandler. It completely changes the behavior and forces users that even for managed resources always check all the ReconcileResult. I'm fine to postpone the release, but this needs to be thinked though, samples made, etc. Because that is completely different behavior and not seamless / drop in replacement at all as in original design.

4. Again the fail fast approach is I think right in this case, if there is a problem with the Secret reconciliation, it should not reconcile further but retry. Since other resources depend on it. This was changed last week, we should rather revert this behavior, because actually makes issues like this.

[csviri]

people want to be able to make decisions based on the reconcile result of dependents and right now, the reconcile result doesn't cover all the possibilities

There is no decision to make in this case. It is basically should be just retries if there is an issue with secret creation. Can you describe actual use case when people want to make decision bases on error? and how would that look like?

So in this case the actual thing we want to do just retry if the secret creation fails, don't go further.
Event if there is an error status in ReconcileResult not sure how that would make the result behavior different at the and.

[metacosm]

Note that all this discussion is out of scope for the next release but it's still interesting to talk about this problem now.

[metacosm]

people want to be able to make decisions based on the reconcile result of dependents and right now, the reconcile result doesn't cover all the possibilities

There is no decision to make in this case. It is basically should be just retries if there is an issue with secret creation. Can you describe actual use case when people want to make decision bases on error? and how would that look like?

The problem is that the sample currently throws an exception when the secret is not present in the context. When it's not present, it's usually not because we couldn't create it but rather because the associated schema is not ready. Handling this use case via error handling is wrong because it's not an error condition and that might lead to retries being exhausted despite the fact that nothing is actually wrong with the system. To be clear, I'm not saying that this should be fixed for the current scope but throwing an exception in this situation is not the proper way to deal with this because this is not an error, just part of the reconciliation process. Note that this problem wouldn't be solved with an error case on ReconcileResult either, though, because this isn't an error. The problem is that ReconcileResult is currently under-specified and doesn't cover all the possible cases.

[csviri]

The problem is that the sample currently throws an exception when the secret is not present in the context. When it's not present, it's usually not because we couldn't create it but rather because the associated schema is not ready

Will take a look, why it might happen, but if the secret if created and it's not in a context is simply a bug withing the current implementation. Will take a look on this because this actually should not happen, since we putting the resource into the event source if we created it)

However @scrocquesel clearly saying in the issue:

Actually, when the secret is not yet created, the mysql schema sample is throwing some errors

[csviri]

@scrocquesel could you please describe more in detail the actual situation what is happening regarding the secret, so it is really not created at that point, or how is this happening?