evaluator: 0.5 regression with disjunctions and comprehensions

[slewiskelly]

What version of CUE are you using (cue version)?

$ cue version
cue version v0.5.0-beta.2

       -compiler gc
        -ldflags -s -w
     CGO_ENABLED 0
          GOARCH arm64
            GOOS darwin

Does this issue reproduce with the latest release?

Yes. git bisect pinpoints to 748a685 (https://review.gerrithub.io/c/cue-lang/cue/+/529524).

What did you do?

cue export with the following configuration snippet.

A similar, but larger configuration has been used for several months using v0.4.3. Upon evaluating v0.5.0, failures are encountered.

Reproducer

Foo: #Abstract & {spec: foo: {}}
Bar: #Abstract & {spec: bar: {}}

#Abstract: X={
	spec: _#Spec

	resource: _Thing & {_X: spec: X.spec}
}

_#Spec: *_#SpecFoo | _#SpecBar

_#SpecFoo: {
	foo: {
		min: int | *10
		max: int | *20
	}
}

_#SpecBar: {
	bar: {
		min: int | *30
		max: int | *40
	}
}

_Thing: #Constrained & {
	_X: _

	spec: {
		if _X.spec.foo != _|_ {
			minFoo: _X.spec.foo.min
			maxFoo: _X.spec.foo.max
		}

		if _X.spec.bar != _|_ {
			minBar: _X.spec.bar.min
			maxBar: _X.spec.bar.max
		}
	}
}

#Constrained: #Base & {
	spec: {
		minFoo:  int | *10
		maxFoo:  int | *20
		minBar?: null
		maxBar?: null
	} | {
		minBar:  int | *30
		maxBar:  int | *40
		minFoo?: null
		maxFoo?: null
	}

	spec: *{
		fuga?: null
	} | {
		hoge?: null
	}
}

#Base: {
	spec: {
		minFoo?: null | int
		maxFoo?: null | int
		minBar?: null | int
		maxBar?: null | int

		hoge?: null | bool
		fuga?: null | bool
	}
}

Forgive the length of the reproducer. I tried to maintain the spirit of what is trying to be accomplished, while removing as much as possible.

What did you expect to see?

Result v0.4.3

{
    "Foo": {
        "spec": {
            "foo": {
                "min": 10,
                "max": 20
            }
        },
        "resource": {
            "spec": {
                "minFoo": 10,
                "maxFoo": 20
            }
        }
    },
    "Bar": {
        "spec": {
            "bar": {
                "min": 30,
                "max": 40
            }
        },
        "resource": {
            "spec": {
                "minBar": 30,
                "maxBar": 40
            }
        }
    }
}

What did you see instead?

Result v0.5.0

Bar.resource.spec: 6 errors in empty disjunction:
Bar.resource.spec.minBar: 2 errors in empty disjunction:
Bar.resource.spec.minBar: conflicting values null and int (mismatched types null and int):
    ./test.cue:2:6
    ./test.cue:7:12
    ./test.cue:26:9
    ./test.cue:42:15
    ./test.cue:46:12
    ./test.cue:66:19
Bar.resource.spec.minFoo: 2 errors in empty disjunction:
Bar.resource.spec.minFoo: conflicting values null and 10 (mismatched types null and int):
    ./test.cue:2:6
    ./test.cue:7:12
    ./test.cue:14:15
    ./test.cue:30:3
    ./test.cue:31:12
    ./test.cue:51:12
Bar.resource.spec.minFoo: conflicting values null and int (mismatched types null and int):
    ./test.cue:2:6
    ./test.cue:7:12
    ./test.cue:26:9
    ./test.cue:42:15
    ./test.cue:51:12
    ./test.cue:64:19
Bar.resource.spec.minBar: undefined field: min:
    ./test.cue:36:24

I can make it work with by making (at least) either of the following two changes:

1. Writing the equivalent using a combination of a conjunction and disjunction, without the use of additional definitions.

Diff

@@ -2,10 +2,18 @@ Foo: #Abstract & {spec: foo: {}}
 Bar: #Abstract & {spec: bar: {}}

 #Abstract: X={
-	spec: _#Spec
-
 	resource: _Thing & {_X: spec: X.spec}
-}
+} & (*{
+	spec: foo: {
+		min: int | *10
+		max: int | *20
+	}
+} | {
+	spec: bar: {
+		min: int | *30
+		max: int | *40
+	}
+})

 _#Spec: *_#SpecFoo | _#SpecBar

2. Removing a disjunction within the #Constrained definition

Diff

@@ -51,12 +51,6 @@ _Thing: #Constrained & {
 		minFoo?: null
 		maxFoo?: null
 	}
-
-	spec: *{
-		fuga?: null
-	} | {
-		hoge?: null
-	}
 }

[mvdan]

Reduced from 72 lines to 25, though note that now master only gives one of the errors. If we have a potential fix, we should verify if the other errors are still present.

$ cat repro.cue
Foo: #Obj & {spec: foo: {}}
Bar: #Obj & {spec: bar: {}}

#Obj: X={
	spec: *#SpecFoo | #SpecBar

	out: #Out & {
		_Xspec: X.spec
		if _Xspec.foo != _|_ {
			minFoo: _Xspec.foo.min
		}
		if _Xspec.bar != _|_ {
			minBar: _Xspec.bar.min
		}
	}
}

#SpecFoo: foo: min: int | *10
#SpecBar: bar: min: int | *20

#Out: {
	{minFoo: int} | {minBar: int}

	*{nullFoo: null} | {nullBar: null}
}
$ cue export repro.cue 
Bar.out.minBar: undefined field: min:
    ./repro.cue:13:23

This was with cue version 8fb13d4. cc @mpvl

[mvdan]

Worth noting that the workarounds that @slewiskelly mentions still appear to work, such as removing the *{nullFoo: null} | {nullBar: null} line.

[mpvl]

Unfortunately, the fix was breaking too much other code. We already knew this was a brittle change.

We now plan to fix this as part of an evaluator revamp. This has very high priority, but will not make it to v0.5, unfortunately.

[bttk]

@mpvl: How should we understand the status of the evaluator rework? If I have a regression in my project when I go from v0.4.3 to v0.5.0-beta.5 similar to the ones I have reported before, should I try to put together a repro?

[myitcv]

@bttk - we will be publishing a plan outlining next steps for v0.5.0, v0.6.0 and v0.7.0 this week. I will link to that discussion from here.

[mpvl]

@bttk: repros are definitely useful. Another useful thing to do would be to add your project to Unity. That way we can catch any issues before we release the new evaluator.

We are in the process of supporting private repos. Please contact @mvdan for early access.

[myitcv]

@slewiskelly @bttk - we will shortly be posting a discussion about how we are approaching the v0.5.0, v0.6.0 and v0.7.0 releases. Indeed we be covering this topic on the Community Call next week.

[myitcv]

Marking this as v0.7.0. If we were to address this properly we would likely need an involved and extensive fix as part of v0.6.0. As things stand, we don't want to delay v0.6.0 if we can avoid it, and instead prefer to address this along with other performance and disjunction issues in v0.7.0. This has the added benefit of the fix likely being considerably easier, because of refactors in the evaluation engine as part of v0.7.0 (that are not part of v0.6.0, where the cost of making such changes is higher).

[slewiskelly]

Might the fix be included in the v0.8 release?

We'd appreciate to be able to pick-up the release, as we've been very much looking forward to the performance improvements, as well as adopting some other features since v0.4.

[myitcv]

@slewiskelly - apologies for the delay in replying here. Since your message we've been improving how we communicate about updates to the performance of CUE via #2857 and an office hours session on Thursday #2862.

Next week we will check where this issue stands with regards to the new evaluator, and whether it makes sense for a fix to be applied to the old (current) evaluator.

[myitcv]

@slewiskelly - not forgotten this issue. Marcel is still deep in work on performance in the new evaluator (#2884), and per the updates in #2850 we are targeting releasing an experimental version of the new evaluator via #2886. That was scheduled to happen by the end of March, but we've slipped a bit on that. We'll be posting an update to #2850 today updating that we should be in a position to release an experimental version next week.

At which point we will be able to return to this issue and properly evaluate whether a fix can be included to the "old" evaluator (so named, simply because we need a way of distinguishing the two).

I've placed this issue in the Evaluator Roadmap as "planned" to indicate that we will look at whether it's feasible to include a fix in the v0.8 series for the regression.

[mvdan]

Here is an updated testscript that passes - it shows the current behavior as of 6694c55.

It's worth noting that evalv3 still fails, although with a different error now.

! exec cue export input.cue
cmp stderr evalv2.stderr

env CUE_EXPERIMENT=evalv3
! exec cue export input.cue
cmp stderr evalv3.stderr

-- evalv2.stderr --
Bar.out.minBar: undefined field: min:
    ./input.cue:13:23
-- evalv3.stderr --
Bar.out: incomplete value {_Xspec:~(Bar.spec),minFoo:int,nullFoo:null} | {_Xspec:~(Bar.spec),minBar:int,nullFoo:null}
Foo.out: incomplete value {_Xspec:~(Foo.spec),minFoo:int,nullFoo:null} | {_Xspec:~(Foo.spec),minBar:int,nullFoo:null}
-- input.cue --
Foo: #Obj & {spec: foo: {}}
Bar: #Obj & {spec: bar: {}}

#Obj: X={
	spec: *#SpecFoo | #SpecBar

	out: #Out & {
		_Xspec: X.spec
		if _Xspec.foo != _|_ {
			minFoo: _Xspec.foo.min
		}
		if _Xspec.bar != _|_ {
			minBar: _Xspec.bar.min
		}
	}
}

#SpecFoo: foo: min: int | *10
#SpecBar: bar: min: int | *20

#Out: {
	{minFoo: int} | {minBar: int}

	*{nullFoo: null} | {nullBar: null}
}