Differing behaviour between cue eval and yaml.Unmarshal and hidden fields?

[cueckoo]

Originally opened by @edwardgeorge in cuelang/cue#976

I was testing how hidden fields behave when originating from yaml files and I'm confused by the differing behaviour between these two examples. I would expect them to give the same output but using yaml.Unmarshal gives an error.

foo.yaml:

data:
  _invisible: ["a", "b"]

foo.cue:

package foo

data: {
    _invisible: [...]
}
not_invisible: data._invisible

the output of cue eval foo.cue foo.yaml is:

data: {}
not_invisible: ["a", "b"]

(cue version 0.4.0-beta.1 darwin/amd64)

However when I attempt the following I get an error.

foo_tool.cue:

package foo

import (
    "encoding/json"
    "encoding/yaml"
    "tool/cli"
    "tool/file"
)

filename_str: *"foo.yaml" | string

command: example: {
    eval: file.Read & {
        filename: filename_str,
        contents: bytes,
    }
    let res = yaml.Unmarshal(eval.contents) & {
        data: {
            _invisible: _
        },
        not_invisible: data._invisible
    }
    out: cli.Print & {
        text: json.Marshal(res)
    }
}

the output is:

❯ cue cmd example foo_tool.cue
command.example.out.text: invalid string argument: cannot convert incomplete value "_" to JSON:
    ./foo_tool.cue:23:5
    ./foo_tool.cue:24:9
    tool/cli:4:3

I would have expected them both to behave the same. is there something I'm misunderstanding here? or am I doing something wrong? (I'm obviously new to cue)

[cueckoo]

Original reply by @verdverm in cuelang/cue#976 (comment)

I expected _label fields in YAML to be regular fields like "_hidden": value, given yaml has no concept of hidden fields.

The following is an interesting example, perhaps meaning the parser needs to be adjusted?

data.yaml

data:
  _hid: foo
  "_hid": 1

data.cue

data: {
        _hid: string
        "_hid": int
}

---

Not sure the exact answer to your original question, could be some interplay of the YAML parser(s), hidden fields, and/or when the values are unified in the tool subsystem

[cueckoo]

Original reply by @mpvl in cuelang/cue#976 (comment)

Yes @verdverm, you're correct. This seems like a YAML conversion bug. So the tool example is correct, but the first incorrect.

[cueckoo]

Original reply by @mpvl in cuelang/cue#976 (comment)

I've sent out a fix for this: https://cue-review.googlesource.com/c/cue/+/9841.

[cueckoo]

Original reply by @mpvl in cuelang/cue#976 (comment)

Just FTR:

CUE supports multiple field types. The type that YAML and JSON supports are called regular fields in CUE. In summary, CUE supports:

field syntax	description
"foo"	regular fields are the "data" fields, basically the fields that you would find in YAML or JSON. The double quotes may be omitted before a : if the field name is a valid identifier that does not start with a _.
_foo	hidden fields are only visible within the package in which they are defined. They are not exported as data.
#foo	definitions are also not exported as data, but unlike hidden fields, they are visible within a package. They are typically used to define CUE values that are constants or templates. CUE values defined with definitions are also closed by default.
_#foo	hidden definitions are like definitions, but are only visible within the package in which they are defined.

Note that identifiers may start with $. These are sometimes used to indicate special kind of fields in JSON or YAML, but have no special meaning in CUE.