Request fails with JSON files containing escape sequences for line feed codes

[k2tzumi]

If you try to send a request for a json file that contains a newline code in the json value, the json itself will be sent as a string.

• json files containing line feed codes
  https://github.com/k2tzumi/runn/blob/5cbb9f68afb2bc3344ec5aec094755ddeaa4f3dd/testdata/newline.json

• runbook that calls the json file
  https://github.com/k2tzumi/runn/pull/20/files#diff-529bf6e17bec3f0713aa61973fdc8ff6ad93c059f1ba1650a90e498e582ab934

• result
  https://github.com/k2tzumi/runn/actions/runs/4901420664/jobs/8752680939#step:7:132

• debug
  json body is not json!

  $ go run cmd/runn/main.go run testdata/book/httpbin_include.yml --debug
  .. snip ..
  
  Run 'req' on 'testing include'.steps[1]
  -----START HTTP REQUEST-----
  POST /post?count=0 HTTP/1.1
  Host: httpbin.org
  Content-Type: application/json
  X-Test: default
  
  "{\"bar\":\"abc\\r\\ndef\",\"foo\":\"abc\\ndef\"}"
  -----END HTTP REQUEST-----
  -----START HTTP RESPONSE-----
  HTTP/2.0 200 OK
  Content-Length: 581
  Access-Control-Allow-Credentials: true
  Access-Control-Allow-Origin: *
  Content-Type: application/json
  Date: Sat, 06 May 2023 12:17:56 GMT
  Server: gunicorn/19.9.0
  
  {
    "args": {
      "count": "0"
    }, 
    "data": "\"{\\\"bar\\\":\\\"abc\\\\r\\\\ndef\\\",\\\"foo\\\":\\\"abc\\\\ndef\\\"}\"", 
    "files": {}, 
    "form": {}, 
    "headers": {
      "Accept-Encoding": "gzip", 
      "Content-Length": "50", 
      "Content-Type": "application/json", 
      "Host": "httpbin.org", 
      "User-Agent": "Go-http-client/2.0", 
      "X-Amzn-Trace-Id": "Root=1-64564571-525c7cd93ad52c044831da9c", 
      "X-Test": "default"
    }, 
    "json": "{\"bar\":\"abc\\r\\ndef\",\"foo\":\"abc\\ndef\"}", 
    "origin": "20.205.207.201", 
    "url": "https://httpbin.org/post?count=0"
  }
  
  -----END HTTP RESPONSE-----
  Run 'test' on 'testing include'.steps[1]
  F
  
  1) t/b/httpbin_include.yml
    Failure/Error: test failed on 'testing include'.steps[1]: (current.res.status == 200
    && diff(current.res.body.json, vars.externalJsonRequestBody) == "") is not true
    current.res.status == 200
    && diff(current.res.body.json, vars.externalJsonRequestBody) == ""
    ├── current.res.status => 200
    ├── 200 => 200
    ├── diff(current.res.body.json, vars.externalJsonRequestBody) => "  any(
    │   -         string(`{"bar":"abc\r\ndef","foo":"abc\ndef"}`),
    │   +         map[string]any{"bar": string("abc\r\ndef"), "foo": string("abc\ndef")},
    │     )
    │   "
    ├── current.res.body.json => "{"bar":"abc\r\ndef","foo":"abc\ndef"}"
    ├── vars.externalJsonRequestBody => {"bar":"abc\r\ndef","foo":"abc\ndef"}
    └── "" => ""
  
  1 scenario, 0 skipped, 1 failure

• For json without newline code
  Essentially, it's correct to leave the json

  $ go run cmd/runn/main.go run testdata/book/httpbin_include.yml --debug
  .. snip ..
  
  Run 'req' on 'testing include'.steps[1]
  -----START HTTP REQUEST-----
  POST /post?count=0 HTTP/1.1
  Host: httpbin.org
  Content-Type: application/json
  X-Test: default
  
  {"bar":1,"foo":"test"}
  -----END HTTP REQUEST-----
  -----START HTTP RESPONSE-----
  HTTP/2.0 200 OK
  Content-Length: 523
  
  .. snip ..

[k1LoW]

@k2tzumi Thank you!

[k1LoW]

I noticed a difference in the handling of carriage return ( \r ) in goccy/go-yaml while working on this issue.

goccy/go-yaml#371

[k1LoW]

I found the cause of this bug.

When a map variable is given as follows,

post:
    body:
        application/json: {{ vars.someMap }}

It seems that it happened to work correctly only if it does not contain newlines (\n).

This seems to be the current behavior of k1LoW/expand.

https://github.com/k1LoW/expand/blob/dd65a4a0fe0484196290c97edd24579b9832ac38/expand.go#L63-L65

It seems to need some adjusting for everything to work well.

[k1LoW]

Fixed at #497.

Also, we will continue to watch goccy/go-yaml's handling of \r encoding.