Proposal: predicate builtins

[mpvl]

Proposal: predicate builtins

Summary

Add a family of predeclared builtins that let CUE inspect a value's state — whether a field is present, whether an expression is valid, whether it is concrete, and whether a label is permitted — returning a boolean without propagating errors. They are first-class, well-defined replacements for the long-standing comparison-to-bottom idiom (!= _|_ / == _|_).

The set:

builtin	kind	question
exists(ref)	function → bool	is the referenced field present as a regular member?
allows(s, label)	function → bool	may label (string field / int index) appear on s?
isValid(expr)	function → bool	does expr evaluate without a permanent error?
isConcrete(expr)	function → bool	does expr evaluate to a concrete value?
existsN(n, refs…)	validator	do n of the referenced fields exist?
validN(n, exprs…)	validator	are n of the expressions valid?
concreteN(n, exprs…)	validator	are n of the expressions concrete?

In every …N validator, n is the same kind of count constraint: a literal requires exactly that many, and a bound such as >=2 or <=1 requires at least or at most that many.

They are gated behind a new per-file experiment, @experiment(predicates) (preview in v0.17.0). The normative semantics and worked examples are in the language spec (doc/ref/spec.md).

This supersedes the relevant part of #943, with the other builtins it mentioned spun out separately (e.g. must is already tracked in #575).

Motivation

Users routinely need to branch on properties that CUE's monotonic, error-propagating evaluation otherwise hides — is this field present? did this evaluate without error? is it concrete yet? Today this is written as a comparison to bottom:

if x != _|_ { out: x }   // "x is present / valid"

That idiom is problematic:

• It conflates three distinct questions — presence, validity, and concreteness.
• It has no documented semantics; the behavior is an artifact of the implementation.

Giving each question its own named builtin makes the intent explicit and the semantics specified, and lets the evaluator surface order-dependence as a cycle rather than a silent wrong answer.

A concrete driver is protobuf's oneof — "exactly one of these fields is set." That is currently awkward to encode in CUE (it needs a disjunction of structs or hand-written constraints); existsN(1, a, b, c) expresses it directly. The count validators generalize this "N of M" pattern to presence (existsN), validity (validN), and concreteness (concreteN).

Proposal

All builtins are predeclared and gated behind @experiment(predicates).

Functions (→ bool)

exists(ref) — whether the referenced field is present as a regular member. ref must be a reference (field reference, selector, or index expression); a non-reference argument is an error.

a: {
	x:  1
	y?: string
}
#A: c: 1
exists(a.x)            // true
exists(a.y)            // false: y is optional and unset
exists(a.z)            // false
exists(a["x"])         // true
exists(#A.nonExisting) // false: #A does not allow nonExisting

allows(s, label) — whether label (a string field name or integer index) may appear on the struct/list s. It is the structural "may ever appear" counterpart to exists, and the two compose: allows(s, "k") && exists(s.k).

s: {a: 1}
allows(s, "b")              // true: s is open
allows(close({a: 1}), "b")  // false: closed, "b" not allowed
allows([1, 2, 3], 3)        // false: index out of range

isValid(expr) — true iff expr does not evaluate to a permanent error; an incomplete value that has no internal inconsistencies and could still be made concrete is valid.

isValid(1 + 2)     // true
isValid(1 / 0)     // false: division by zero
isValid("a" & 1)   // false: conflicting values
isValid({a: 1}.b)  // false: field does not exist

isConcrete(expr) — whether expr is concrete (recursively for structs and lists; only regular fields are considered).

isConcrete(1)            // true
isConcrete(int)          // false: int is a type
isConcrete({a: int})     // false: a is not concrete
isConcrete({a?: int})    // true: only regular fields are considered
isConcrete([1, int, 3])  // false: element is not concrete

Count validators

existsN, validN, and concreteN are embedded in a struct to constrain how many of their arguments satisfy a property. All three share the same shape and count semantics: the first argument is the count — a literal n requires exactly n, while a bound such as >=2 or <=1 requires at least or at most that many — and the rest are references (existsN) or expressions (validN, concreteN). existsN directly encodes protobuf's oneof and its relatives:

#OneOf: {
	existsN(1, a, b, c)  // exactly one of a, b, c
	a?: int
	b?: string
	c?: bool
}
foo: #OneOf & {a: 1}          // {a: 1, b?: string, c?: bool}
bad: #OneOf & {a: 1, b: "x"}  // _|_: two fields set

#AtMostOne: {existsN(<=1, x, y), x?: int, y?: int}
#AtLeastTwo: {existsN(>=2, a, b, c), a?: int, b?: int, c?: int}

validN and concreteN are the analogues for validity and concreteness:

value: "hello" & validN(>=1, int & value, string & value)  // ok: string matches
check: {a: 1, b: int, c: "hello", concreteN(>=2, a, b, c)} // ok: a and c concrete

If a count constraint can never be satisfied, validation fails immediately; otherwise it stays incomplete until a definitive answer can be determined.

Open question

What does isValid close over?

isValid(expr) is valid if expr can be made concrete, without error, by making it more specific. For a self-contained expression that is unambiguous. But when the argument is a reference, the value it ultimately takes can depend on parts of the configuration outside the isValid call:

a: c
c?: int
x: isValid(a)

a resolves through c, which a sibling c: 5 could still make concrete. The two readings give different results:

• if concretization reachable through the rest of the configuration counts, isValid(a) is true (a sibling c: 5 can still make a concrete);
• if a is judged only as it resolves within the call, where c is an unsupplied optional, isValid(a) is false.

This needs to be decided and written into the isValid definition; the same question applies to validN.

What does isConcrete report for an optional or required field?

A referenced field may be optional or required rather than a present regular member:

a?: 123
x: isConcrete(a)

The value 123 is concrete, yet a is not present.
Two readings are possible: isConcrete(a) is false (pragmatically, a field that is not present is not concrete), or it is an incomplete error (it is unclear what "concrete" should even mean for a field that does not yet exist). Either way, and unlike exists, which simply answers false for an absent field, isConcrete arguably requires the field to exist before its concreteness is a meaningful question. Also, if a regular field with the value referred to by isConcrete has an incomplete error, that is an error. The same should hold for the top level. At least these two cases should be consistent. The same applies to concreteN.

Relationship to #943

#943 is stale (it predates allows and isValid), too broad (it also covers must, range, …), and has accumulated unrelated discussion. This proposal pins down the concrete, implemented design for the presence/validity/concreteness predicates, and deliberately uses the name predicates rather than "reflect"/"introspection" — reserving the latter for possible future type or schema reflection.

[seh]

The assertion that optional fields don't exist is hard to accept. This area of CUE is still hard to understand. My coworkers have asked me several times recently what it means for an optional field to have a concrete value, or to have a default value.

My initial take is that isConcrete evaluated against an optional field should report whether the value held by that field is concrete.

You shared this example:

a?: 123
x: isConcrete(a)

I read that and think, "a is 123, which is a concrete value", without paying any attention to whether a is optional or not. We can augment that example slightly, without changing a's value, and ask the same question of it again:

a?: 123
a: _ // Effectively no statement, other than that 'a' should be exported.
x: isConcrete(a)

Now, you could say, "Well, that's a different question; you're asking about a different a there—a present, regular a."

Again, I read the isConcrete predicate to be asking not about qualification or presence of the expression, but rather about the value that it bears. Does an optional field bear a value or not?

[seh]

That I have trouble with the terms here is more indication that we got optional and required fields, but without a firm enough grounding in how to both discuss and use them.

I'm confused by this statement of yours:

would imply that isConcrete({a?: int}) should be false, which is clearly wrong

If isConcrete({a?: int}) is clearly not false, are you suggesting that instead it's true, or that it should result in an error, because the question itself is malformed?

[mpvl]

isConcrete({a?: int}) should be true, as the optional field plays no role in whether a value is concrete.

So similarly, it would be surprising to me if isConcrete would behave differently when directly referencing an optional field directly instead of indirectly. But I realize evaluating an argument to be passed to isConcrete is somewhat of an orthogonal concern.

Having better language around regular fields versus "schema" fields would certainly be useful. CUE derives its power from defining both within the same space. But I understand that can also be confusing. But basically, the way I hold it: regular fields -> JSON, constraint fields -> JSON Schema. They just happen to be able to be in the same object.

[mvdan]

@seh another way to think about Marcel's isConcrete({a?: int}) is that cue vet -c accepts it and cue export renders it as {}. So it is a concrete struct, because it finalizes and exports as an empty struct with zero fields.

[seh]

Reading both of your responses now, I see that the question mark on a renders it "invisible" for this question, such that {a?: 123} and {} are equivalent treatments of the argument to isConcrete here.

Does that mean that "What does isConcrete report for an optional field?" is still an open question, or did we just settle it here? Perhaps isConcrete({a?: 123}) is different enough from isConcrete(a), because the former is looking at a struct value, not a reference to a field.

[mvdan]

Indeed the two cases are different. Given the CUE {a?: 123}, you can export the whole value just fine as {}. But you cannot export the reference a, because resolving that reference results in a "cannot reference optional field" error.

[mxey]

Does this provide a way to check for an optional field having a value, but raising an error if that field can never exist? There is the combination of allows & exists, but what I am thinking of is how to avoid typos when using exists.

[mpvl]

That was the original implementation of exists, and either interpretation would be fine.

In the end we ended up with the current semantics. But as said, this may change based on popular demand.

There were several reasons to take this stance:

• Using a stricter semantics makes it almost impossible to auto-rewrite comparing to bottom.
• You can get the semantics you want roughly already with try
• We can still do the typo checking in some cue linter (cue lint or an extended cue vet).

But as said, this is a bit of a toss up.

[mpvl]

One option is to have an option exists(x.b, strict: true) down the line.

[mxey]

You can get the semantics you want roughly already with try

Do you mean by using an optional field in a try? That won't swallow errors?

[myitcv]

Some feedback from a full-scale adoption exercise. helm2cue generates CUE from Helm charts, and its output leans heavily on the historical != _|_ idiom this proposal replaces: a _nonzero truthiness helper that type-dispatches over six kinds, a _typeof helper, nested-path traversal (dig), key-membership tests (hasKey, merge/mergeOverwrite), and type tests (kindIs, typeIs). I converted every one to the predicate builtins behind @experiment(predicates), round-trip verified against helm template (cue-exp/helm2cue#127).

Overall: it works, and the output reads far better than the bottom comparisons. But having looked harder at why it works, my headline finding has changed: the drop-in only works because isValid returns what I think is the wrong answer in one case, and helm2cue's generated code is unwittingly depending on that. The deeper cause is that exists — the primitive whose spelling fits these sites — is unusable for these patterns, so the conversion was forced onto isValid everywhere.

Each claim below is a self-contained testscript reproducer, run with cmd/testscript against cue version v0.17.0-alpha.2.0.20260612134833-f88c737c8ff6. Every archive sets @experiment(predicates) and language.version: "v0.17.0". Each reproducer asserts the behaviour I believe is correct (or that a code generator needs), so the intended result is pass. Where the implementation differs, I follow the reproducer with the verbatim testscript run — which fails, and that failure is the finding. (In those run blocks I've elided testscript's echo of the script comment, shown in the archive just above, keeping the command trace and result.)

1. isValid misclassifies an absent/incomplete selection as invalid — I think this is a bug

isValid(x) matched x != _|_ at every site — and that clean uniformity is itself the tell. The sites aren't all the same kind of question. A few are genuine validity checks (isValid(x & struct.MaxFields(0)) as an emptiness test). But several are really presence checks — does this field exist? — and isValid only "covered" them because it happens to return false for an absent reference: an unset optional field; a field set to a reference to an absent field; an index whose base does not resolve. A builtin named for validity ended up answering presence questions across the board, which is what finding 2 is really about. And in the absent case it doesn't merely overreach — it returns the wrong answer.

The proposal describes isValid as true when there is no permanent error, with incomplete-but-completable values valid. An absent-field selection on an open struct is an incomplete error, not a permanent one. The proposal doesn't pin down a single answer here — it could be true, or isValid could itself fail to reduce, or something else — but it does seem fairly clear the answer is something other than false, since false is the definite "this is invalid" verdict and an incomplete value hasn't earned it. The testable property I'm confident in: a bare incomplete selection and the same selection under a type conjunction are both incomplete, so isValid must classify them alike. That reproducer should pass:

# s.absent and s.absent & int are both incomplete selections, so isValid
# must classify them alike; in particular neither is the definite "invalid"
# verdict false.
exec cue export x.cue -e out
cmp stdout stdout.golden

-- cue.mod/module.cue --
module: "mod.example"
language: version: "v0.17.0"
-- x.cue --
@experiment(predicates)

s: present: "x"

out: isValid(s.absent) == isValid(s.absent & int)
-- stdout.golden --
true

What we actually see when we run it:

$ testscript r1-expected.txtar
> exec cue export x.cue -e out
[stdout]
false
> cmp stdout stdout.golden
diff stdout stdout.golden
--- stdout
+++ stdout.golden
@@ -1,1 +1,1 @@
-false
+true

FAIL: r1-expected.txtar:5: stdout and stdout.golden differ
failed run

The bare-selector branch is the one classifying the incomplete value as false — the value the proposal rules out. That's the bug; it isn't two defensible readings.

Two consequences:

• I'm not asking to pin the false into the spec — nor am I asking for a specific replacement value (true or a non-reducing result both fit the wording). I just think false is the one answer the proposal rules out. I'm flagging it because helm2cue's generated code currently relies on that false (next point), so fixing it will, rightly, force a codegen change. Better that than building a generator on behaviour that contradicts the proposal.
• isValid(val & T) is therefore not a faithful replacement for (val & T) != _|_. Every type test helm2cue emits needs the two-call guard isValid(val) && isValid(val & T) — verbose, and the reason for the leading guard is non-obvious. Worth either blessing that idiom explicitly or offering a direct "present and unifies with T" predicate (a kind-flavoured builtin may serve better than predicates here).

2. The conversion was forced onto isValid as a presence test, because exists can't reach these patterns

This is the real story behind finding 1. Every presence-flavoured site (hasKey, dig, key membership in merge) wants a presence predicate, and exists is exactly the right spelling. In practice I couldn't use it — for one reason in exists itself, and a second that turns out not to be about exists at all, but about how these helpers are encoded.

First, exists is fine on a field or key whose immediate base resolves — an absent field or key yields false, not an error. This reproducer passes:

# When the immediate base resolves, exists answers presence correctly: an
# absent field or key yields false, not an error.
exec cue export x.cue -e out
cmp stdout stdout.golden

-- cue.mod/module.cue --
module: "mod.example"
language: version: "v0.17.0"
-- x.cue --
@experiment(predicates)

vals: str: "x"
m: present: "x"

out: {
	existsAbsentField: exists(vals.absent)       // base resolves -> false
	existsAbsentKey:   exists(m["absent"])       // base resolves -> false
	isValidIdx:        isValid(vals.absent["k"]) // false
}
-- stdout.golden --
{
    "existsAbsentField": false,
    "existsAbsentKey": false,
    "isValidIdx": false
}

But it errors once you index or select through a base that is itself unresolved — the chained pattern (.Values.a.b.c) that dominates config templating, where an intermediate is routinely absent. For presence that should be false, so this reproducer should pass:

# For a presence check, indexing through an absent base means "no" -> false.
# This is the chained-presence case that dominates config templating
# (.Values.a.b.c). Without a concreteness requirement (cue vet -c=false) the
# package should be valid; with one (cue export) it should yield false.
exec cue vet x.cue -c=false
exec cue export x.cue -e out
cmp stdout stdout.golden

-- cue.mod/module.cue --
module: "mod.example"
language: version: "v0.17.0"
-- x.cue --
@experiment(predicates)

vals: str: "x"

out: exists(vals.absent["k"])
-- stdout.golden --
false

What we actually see when we run it — cue vet -c=false (no concrete result required) passes, but cue export (concrete result required) errors instead of yielding false:

$ testscript r3-expected.txtar
> exec cue vet x.cue -c=false
> exec cue export x.cue -e out
[stderr]
out: undefined field: absent:
    ./x.cue:5:18
[exit status 1]
FAIL: r3-expected.txtar:6: unexpected command failure
failed run

An absent base should simply mean "no" for presence. This looks like the job of a non-strict exists (the strict/non-strict distinction mentioned upthread): non-strict letting an unresolved base propagate as false, strict keeping the typo-detecting error.

Second — and this one is really about the function pattern, not exists. As #3711 sets out, a helper's out is written in terms of #arg but isn't tied to it: out can be valid even when #arg is in error. With #f: {#arg?: _, out: 5}, (#f & {#arg: true & false}).out is 5, not an error.

The predicates proposal turns out to supply the tie #3711 found missing — no must(), no change to the error model. Unify a validity validator into the result and out becomes valid iff #arg is:

# The function pattern's result is not tied to its argument: out can be
# valid even when #arg is in error. Unifying a validity validator into the
# result restores the tie — out is valid iff #arg is. A valid argument
# passes through to 5; an invalid one makes out an error.
exec cue export x.cue -e ok
cmp stdout ok.golden

! exec cue export x.cue -e bad
stderr 'validN: 0 matched, expected 1'

-- cue.mod/module.cue --
module: "mod.example"
language: version: "v0.17.0"
-- x.cue --
@experiment(predicates)

#g: {#arg?: _, out: validN(1, #arg) & 5}

ok:  (#g & {#arg: 7}).out
bad: (#g & {#arg: true & false}).out
-- ok.golden --
5

A valid argument passes through to 5; an invalid one makes out an error. With the argument tied this way, exists can then be used as normal to test parts of it.

There's a catch, though, and it loops straight back to finding 1. When #arg is optional, you usually want to validate it only when present — provided ⇒ validated, absent ⇒ result unaffected. The tie doesn't give you that: an absent #arg is counted as not valid, so the validator rejects it rather than passing through. This reproducer should pass:

# An optional argument should be validated only when present: provided ->
# validated; absent -> the result unaffected, here yielding 5. (#arg is the
# validity-tied function argument from r4-tie.)
exec cue export x.cue -e out
cmp stdout out.golden

-- cue.mod/module.cue --
module: "mod.example"
language: version: "v0.17.0"
-- x.cue --
@experiment(predicates)

#g: {#arg?: _, out: validN(1, #arg) & 5}

out: (#g & {}).out
-- out.golden --
5

What we actually see when we run it:

$ testscript r5-optional-gap.txtar
> exec cue export x.cue -e out
[stderr]
out: validN: validN: 0 matched, expected 1:
    ./x.cue:3:21
    ./x.cue:3:28
[exit status 1]
FAIL: r5-optional-gap.txtar:4: unexpected command failure
failed run

That "an absent optional counts as not valid" is exactly finding 1's misclassification (isValid(<absent>) == false) resurfacing through validN. So it isn't a separate gap so much as another face of finding 1 — and it sharpens the ask there: for "validate only if present" to be expressible, an absent/incomplete argument has to resolve as valid (not false, and not merely non-reducing). Which way finding 1 lands decides whether this pattern works at all.

So isValid — which accepts any operand and (via the finding-1 bug) already answers presence for references — was the path of least resistance, and exists went largely unused. The pieces this proposal owns are the two findings: exists stuck on a chained/unresolved base, and the isValid classification of an absent/incomplete selection — the latter governing finding 1 directly and whether the validator-tie above works for optional arguments.

What I'd ask for

These pay off only together:

1. Fix isValid so an incomplete (absent / unresolved) selection is not reported as false — true or a non-reducing result, whichever the proposal settles on — bringing the bare selector into line with the & T branch and making isValid an honest validity predicate. (Finding 2's validator-tie gives a concrete reason to prefer the valid resolution over non-reducing: validating an optional argument only-when-present needs an absent one to count as valid.)
2. Make exists usable for presence: non-strict treatment of an unresolved base (propagate as false, not error).

With both, helm2cue can compute presence at the call site with exists and reserve isValid for validity — the separation this proposal is for. Fixing (1) on its own breaks the current output: the _nonzero truthiness guard is isValid(#arg), relying on today's isValid(<absent>) == false. Once that's corrected the guard can't simply become exists(#arg) (the function-pattern issue above), so presence has to move to the call site — where it needs (2) to survive chained paths. Hence the two want to land together, with a codegen restructure on the helm2cue side.

3. Smaller notes

• !isValid(x) reads fine as the replacement for x == _|_ (kindIs "invalid", typeIs "<nil>").
• I found no use for existsN/validN/concreteN in this domain — the oneof pattern doesn't arise in Helm conversion. No objection; just a datapoint that the singular forms carried all the weight.
• Since the experiment gates on language version v0.17.0, tools that generate CUE must also bump the language version of the modules they emit. Worth a sentence in the proposal: adoption by generators moves the version requirement onto their users' modules.

Against the cited build, the r1-expected, r3-expected, and r5-optional-gap reproducers fail as shown — those failures are the findings — while r2-exists-base-ok and r4-tie pass. The full conversion is in cue-exp/helm2cue#127.

[myitcv]

Note I have heavily updated the reply above, rather than posting a follow-up reply. There were a number of things which (on deeper reading) were wrong in terms of direction.

[myitcv]

I've done another heavy iteration on the wording