Skip to content

Commit

Permalink
[YAML] Trim trailing whitespace from plain scalars
Browse files Browse the repository at this point in the history
In some cases plain scalars are currently parsed with a trailing
newline. In particular this shows up often when parsing JSON files, e.g.
note the `\n` after `456` below:
```
$ cat test.yaml
{
  "foo": 123,
  "bar": 456
}
$ yaml-bench test.yaml -canonical
%YAML 1.2
---
!!map {
  ? !!str "foo"
  : !!str "123",
  ? !!str "bar"
  : !!str "456\n",
}
...
```
The trailing whitespace ends up causing the conversion of the scalar to
int/bool/etc. to fail, causing the issue seen here:
#15877

From reading the YAML spec (https://yaml.org/spec/1.2.2/#733-plain-style)
it seems like plain scalars should never end with whitespace, so this
change trims all trailing whitespace characters from the
value (specifically `b-line-feed`, `b-carriage-return`, `s-space`, and
`s-tab`).

Reviewed By: scott.linder

Differential Revision: https://reviews.llvm.org/D137118
  • Loading branch information
rkayaith committed Feb 10, 2023
1 parent 35537ae commit 32b3f13
Show file tree
Hide file tree
Showing 3 changed files with 27 additions and 2 deletions.
7 changes: 5 additions & 2 deletions llvm/lib/Support/YAMLParser.cpp
Expand Up @@ -2041,8 +2041,11 @@ StringRef ScalarNode::getValue(SmallVectorImpl<char> &Storage) const {
}
return UnquotedValue;
}
// Plain or block.
return Value.rtrim(' ');
// Plain.
// Trim whitespace ('b-char' and 's-white').
// NOTE: Alternatively we could change the scanner to not include whitespace
// here in the first place.
return Value.rtrim("\x0A\x0D\x20\x09");
}

StringRef ScalarNode::unescapeDoubleQuoted( StringRef UnquotedValue
Expand Down
13 changes: 13 additions & 0 deletions llvm/test/YAMLParser/json.test
@@ -0,0 +1,13 @@
# RUN: yaml-bench -canonical %s | FileCheck %s

# CHECK: !!map {
# CHECK: ? !!str "foo"
# CHECK: : !!str "123",
# CHECK: ? !!str "bar"
# CHECK: : !!str "456",
# CHECK: }

{
"foo": 123,
"bar": 456
}
9 changes: 9 additions & 0 deletions llvm/unittests/Support/YAMLIOTest.cpp
Expand Up @@ -96,6 +96,15 @@ TEST(YAMLIO, TestMapRead) {
EXPECT_EQ(doc.foo, 3);
EXPECT_EQ(doc.bar, 5);
}

{
Input yin("{\"foo\": 3\n, \"bar\": 5}");
yin >> doc;

EXPECT_FALSE(yin.error());
EXPECT_EQ(doc.foo, 3);
EXPECT_EQ(doc.bar, 5);
}
}

TEST(YAMLIO, TestMalformedMapRead) {
Expand Down

0 comments on commit 32b3f13

Please sign in to comment.