Non-deduplication of sets

[timothyhinrichs]

When defining a set, we have two options: using a partial document or a set-comprehension. The two are logically equivalent without aggregation, but with aggregation you the two produce different results. The partial document only eliminates duplicates once you ask for the set as a whole at the end of the search; it leaves duplicates in tact during search. The result is that the same value can be included more than once in an aggregate. In contrast, the set-comprehension eliminates duplicates immediately.

The following two code snippets are identical (and are taken from the Terraform library).

# Set-comprehension
resource_types = all {
    all = {y | input[name]; split(name, ".", outs); y = outs[0]}
}

# Partial document
resource_types_aux[y] {
    input[name]
    split(name, ".", outs)
    y = outs[0]}
}
# Force duplicate elimination
resource_types = resource_types_aux

Two options for addressing.

• Improve debuggability so such issues become apparent immediately. This may be difficult because it seems that the time of de-duplication is not something people will naturally think about.
• Eliminate duplicates more proactively. People will use the 'partial-doc set' because it's syntactically easier than the set-comprehension, both to write and to debug. They probably won't think about it as controlling the de-duplication logic. OPA could analyze the policy to identify when de-duplication must occur proactively (since the caller is an aggregate). The downside is that if someone tries to debug by calling lower-level docs without using an aggregate, the results will be different when she calls the lower-level doc using an aggregate.

[tsandall]

@timothyhinrichs do you have any other thoughts on this? I'm tempted to close for now as we this hasn't come up for quite a while and I'm not sure there is a good solution.

[tsandall]

At minimum we need to clearly document the behaviour.

[timothyhinrichs]

During evaluation, each partial set could record the elements found so far and fail when it encounters an element already in the set (perhaps with a better trace error message). Besides being more intuitive, this would also improve performance. Tracing will look different.