Decode csv value with "&" prefix gives null

[kermorgant]

Describe the bug

Field category in csv file with value &Tradition is converted to null value.
Version of yq: 4.44.1
Operating system: linux
Installed via: yay (archlinux)

Input CSV

data1.csv:

code,category
1,&Tradition

data2.json:

[
  {
    "code": 1,
    "category": null
  }
]

Command
The command you ran:

cat data1.csv | yq  -pcsv -r -ojson > data2.json

Expected behavior

data2.json:

[
  {
    "code": 1,
    "category": "&Tradition"
  }
]

Wrapping the value in the original csv file with double quotes is a workaround, but I'm downloading a google spreadsheet and adding these double quotes would require a preliminary process.

Additional debug

➜ cat test.csv | yq  -pcsv -r -ojson --verbose
[DEBUG] 14:06:31 utils.go:208                      processStdInArgs          missing '-', adding it to the end
[DEBUG] 14:06:31 utils.go:174                      maybeFile                 checking '-' is a file
[DEBUG] 14:06:31 utils.go:179                      maybeFile                 error: stat -: no such file or directory
[DEBUG] 14:06:31 utils.go:183                      maybeFile                 result: false
[DEBUG] 14:06:31 utils.go:236                      processArgs               processed args: [-]
[DEBUG] 14:06:31 utils.go:97                       initCommand               Using input format csv
[DEBUG] 14:06:31 utils.go:98                       initCommand               Using output format json
[DEBUG] 14:06:31 expression_parser.go:28           ParseExpression           Parsing expression: []
[DEBUG] 14:06:31 expression_postfix.go:46          ConvertToPostfix          postfix processing currentToken )
[DEBUG] 14:06:31 expression_postfix.go:126         ConvertToPostfix          opstackLen: 0
[DEBUG] 14:06:31 expression_postfix.go:137         ConvertToPostfix          PostFix Result:
[DEBUG] 14:06:31 decoder_csv_object.go:23          Init                      Detected encoding: Unknown
[DEBUG] 14:06:31 decoder_csv_object.go:54          Decode                    : headerRow[code category]
[DEBUG] 14:06:31 decoder_csv_object.go:64          Decode                    Adding contentRow: [1 &Tradition]
[DEBUG] 14:06:31 candidate_node_yaml.go:109        UnmarshalYAML             UnmarshalYAML !!int
[DEBUG] 14:06:31 candidate_node_yaml.go:117        UnmarshalYAML             UnmarshalYAML -  a scalar
[DEBUG] 14:06:31 candidate_node_yaml.go:109        UnmarshalYAML             UnmarshalYAML !!null
[DEBUG] 14:06:31 candidate_node_yaml.go:117        UnmarshalYAML             UnmarshalYAML -  a scalar

[mikefarah]

The issue is that (mainly for legacy reasons) yq tries to parse all the CSV values as individual yaml strings, and a value starting with '&' in yaml is an anchor. You can switch off this behaviour with the --csv-auto-parse=f flag:

yq '.' data.csv --csv-auto-parse=f -oj
[
  {
    "code": 1,
    "category": "&Tradition"
  }
]

[kermorgant]

That didn't work with my version but I just upgraded for 4.44.3 and now it's ok !

Thanks 🙏