Schema default value of object within array is not being applied.

[GrahamDumpleton]

What steps did you take:

Create a directory called resources for resource files.

In resources/00-schema.yaml add:

#@data/values-schema                                                            
                                                                                
---                                                                             
#@schema/type any=True                                                                                           
spec:                                                                           
  staticPasswords:                                                              
  - username: ""                                                                
    email: ""                                                                   
    hash: ""                                                                    
    userID: ""

In resources/01-dummy.yaml add:

#@ load("@ytt:data", "data")                                                    
                                                                                
---                                                                             
staticPasswords:                                                                
#@ for user in data.values.spec.staticPasswords:                                
- username: #@ user.username                                                                                     
  email: #@ user.email                                                          
  hash: #@ user.hash                                                            
  userID: #@ user.userID or "XXX"
#@ end

Create a values.yaml file containing:

spec:                                                                           
  staticPasswords:                                                              
  - email: "admin@example.com"                                                  
    hash: "$2a$10$2b2cU8CPhOTaGrs1HRQuAueS7JTT5ZHsHSzYiFPm1leZck7Mc8T4W"        
    username: "admin"                                                           
    #userID: ""

Run the command:

ytt -f resources/ --data-values-file values.yaml

What happened:

It generates the error:

ytt: Error: 
- struct has no .userID field or method
    in <toplevel>
      01-dummy.yaml:9 |   userID: #@ user.userID or "XXX"

What did you expect:

Expected it to generate:

staticPasswords:
- username: admin
  email: admin@example.com
  hash: $2a$10$2b2cU8CPhOTaGrs1HRQuAueS7JTT5ZHsHSzYiFPm1leZck7Mc8T4W
  userID: XXX

Anything else you would like to add:

The problem is triggered due to the use of:

#@schema/type any=True

on the spec property in the schema file. If you comment out or remove that line you get the desired result.

That is, using a schema file of:

#@data/values-schema                                                            
                                                                                
---                                                                                                                                                                     
spec:                                                                                           
  staticPasswords:                                                              
  - username: ""                                                                
    email: ""                                                                   
    hash: ""                                                                    
    userID: ""

gives the expected result.

Trying to reverse the impact of schema/type any=True by applying schema/type any=False to the nested staticPasswords property doesn't help. That is, following still fails.

#@data/values-schema                                                            
                                                                                
---                                                                             
#@schema/type any=True                                                                                           
spec:                                                                           
  #@schema/type any=False                                                       
  staticPasswords:                                                              
  - username: ""                                                                
    email: ""                                                                   
    hash: ""                                                                    
    userID: ""

Even though schema/type any=True is used at a higher level in settings hierarchy, would have expected default values to still be applied and userID added to object in array even when not supplied.

Note that the original intent here was to not fail when unexpected properties were supplied in spec for various reasons, but still wanted schema type checks to otherwise apply.

For example, didn't want it to fail if was supplied:

spec:                                                                           
  staticPasswords:                                                              
  - email: "admin@example.com"                                                  
    hash: "$2a$10$2b2cU8CPhOTaGrs1HRQuAueS7JTT5ZHsHSzYiFPm1leZck7Mc8T4W"        
    username: "admin"                                                           
    #userID: ""                                                                 
  unexpectedField: xxx  

which would otherwise yield:

ytt: Error: Overlaying data values (in following order: additional data values): 
One or more data values were invalid
====================================

values.yaml:
    |
  7 |   unexpectedField: xxx
    |

    = found: unexpectedField
    = expected: (a key defined in map) (by 00-schema.yaml:6)
    = hint: declare data values in schema and override them in a data values document

Environment:

• ytt version (use ytt --version):

$ ytt --version
ytt version 0.37.0

• OS (e.g. from /etc/os-release):

$ uname -a
Darwin xxx 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64 x86_64

cc @jorgemoralespou

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

[pivotaljohn]

First off, thank you for this thorough treatment of the subject, @GrahamDumpleton. 🙏🏻

TL;DR:

• this use-case is intended to be supported by @schema/key; unfortunately, that annotation is not yet implemented.
• until then, a workaround is to implement defaulting in a function that wraps the target data value.
• there is a bug where @schema/type any=True is intended to short-circuit any other type definitions within that fragment, but it does not.
• otherwise, @schema/type any=True is working as intended — but that intention isn't communicated well, today.

---

The Current Plan

The plan on the books would be to solve this use-case with schema looking something like this:

#@data/values-schema                                                                                                                                             
---                                                                             
spec:
  staticPasswords:
  - username: ""
    email: ""
    hash: ""
    userID: ""
  #@schema/key allowed=".*" expects="0+"
  #@schema/type any=True
  _:

where:

• @schema/key ... clarifies two things about the key of _:
  • the key can be any valid YAML key
  • there can be any number of such map items
• @schema/type ... clarifies that the value of _: can by anything

However, @schema/key is currently not implemented.

A Workaround

Until then, one way to achieve the desired behavior is to wrap it in a function that defaults the values.

#! data-helper.lib.yml

#@ def staticPassword(user):
username: #@ user.username
email: #@ user.email
hash: #@ user.hash
userID: #@ user.userID if "userID" in user else "XXX"
#@ end

which can be then used to ensure the output contains all the desired content:

#@ load("@ytt:data", "data")
#@ load("data-helper.lib.yml", "staticPassword")
---                                 
staticPasswords:
#@ for user in data.values.spec.staticPasswords:
- #@ staticPassword(user)
#@ end

Clarifying @schema/type any=True

When an author annotates a section of schema as @schema/type any=True, this means that any valid YAML could appear within.

If the annotated node has contents (in the original example, the fragment under spec:), the entire fragment is taken as a literal, singular, whole default value.
ytt stops processing any @schema/... annotations within such fragments (as you noted above).

Enhancement: Describe better the behavior of @schema/type any=True

Two parts:

• the reference documentation for this annotation does not currently clearly explain the behavior described above; it oughta:
  Clarify the intent/meaning/behavior of @schema/type any=True in reference documentation #516
• today @schema/... annotations within the annotated fragment are ignored; in order to better inform the author, such schema ought to be reported as invalid and could further support authors with clarifying hints:
  @schema/... annotations within a subtree annotated @schema/type any=True should be invalid #522

Bug: Array values within a @schema/type any=True should allow any number of items

While investigating this report, I found that with a fragment annotated with @schema/type any=True, ytt continues to verify that contained array values are schema-valid and it should not.

For example, in Schema, an array value must have exactly one (1) item. However, currently, if such a fragment has an array with more than one item (or none) this results in an invalid schema error.

However, as described above, the such fragments are no longer contributing to the schema other than to be a default value and should not be subject to such checks:

• Within schema, fragments annotated with @schema/type any=True are erroneously validated as schema (rather than as a default value) #523