Performance: closedness algorithm consuming lots of memory

[myitcv]

By default, structs are open to adding fields (and we have plans to similarly make lists open by default). That means the following is valid:

x: {name: "CUE"} & {address: "space"}

Structs and lists (containers) can be closed to prevent additional fields/values from being declared. For example, the follow results in an error:

y: close({name: "CUE"}) & {address: "space"}  // ERROR: y.address: field not allowed

Structs and lists can be closed via the close() builtin or via definitions.

Closedness is a powerful aspect when it comes to validation in CUE. However it very much complicates the evaluation process. In a number of the larger configurations we have seen, closedness has resulted in significant memory pressure (and as a result CPU pressure). This pressure can also grow when a closed value (e.g. a definition) is referenced many times from lots of different contexts.

The new evaluator solves this by allowing closedness related structures to be defined and reclaimed locally. In addition, it is more structured about grouping related values, which, in turn, allows it to detect failure and eliminate invalid disjuncts much earlier.

This performance sub-issue captures details and narrative specific to closedness-related performance issues. We will post updates and commentary related to this topic below.

The umbrella performance issue captures higher-level performance updates.

Existing closedness-related bug reports/issues

• …