Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor for optimization #356

Merged
merged 8 commits into from
Jun 5, 2024
Merged

Refactor for optimization #356

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
f1cbccb
optimized basic_node's destructor
fktn-k Jun 4, 2024
3fac0ee
optimized the scanning of block style scalars
fktn-k Jun 4, 2024
3c9b5b6
removed redundant codes in the input_adapter module
fktn-k Jun 4, 2024
0b42b24
improved the filtering of special scalars
fktn-k Jun 4, 2024
7acec11
simplified too classified member variables of the position_tracker class
fktn-k Jun 4, 2024
d953645
updated parse performance data
fktn-k Jun 4, 2024
1b6d23d
run clang-format & amalgamation
fktn-k Jun 4, 2024
e28350d
add test case for missed branch in the scalar_scanner test
fktn-k Jun 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ html-coverage: lcov-coverage
bm-debug:
cmake -B build_bm_debug -S . -DCMAKE_BUILD_TYPE=Debug -DFK_YAML_RUN_BENCHMARK=ON
cmake --build build_bm_debug --config Debug
./build_bm_release/tool/benchmark/benchmarker ./tool/benchmark/macos.yml > ./tool/benchmark/result_debug.log
./build_bm_debug/tool/benchmark/benchmarker ./tool/benchmark/macos.yml > ./tool/benchmark/result_debug.log

bm-release:
cmake -B build_bm_release -S . -DCMAKE_BUILD_TYPE=Release -DFK_YAML_RUN_BENCHMARK=ON
Expand Down
18 changes: 9 additions & 9 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,17 +88,17 @@ $ ctest -C Debug --test-dir build --output-on-failure
## Benchmarking

Though experimental, benchmarking scores are now available with [the dedicated benchmarking tool](./tool/benchmark/README.md) for the parsing.
On an AMD Ryzen 7 5800H @3.20GHz with g++11.4.0 in Ubuntu22.04 (WSL2), fkYAML processes [the YAML source](https://github.com/fktn-k/fkYAML/blob/develop/tool/benchmark/macos.yml) at a competitive speed compared against other existing YAML libraries for C/C++:
On an AMD Ryzen 7 5800H @3.20GHz with g++11.4.0 in Ubuntu22.04 (WSL2), fkYAML parses [the YAML source](https://github.com/fktn-k/fkYAML/blob/develop/tool/benchmark/macos.yml) at a competitive speed compared against other existing YAML libraries for C/C++:

| Benchmark | Release (MB/s) | Debug (MB/s) |
| ---------------------------------- | -------------- | ------------ |
| fkYAML | 35.072 | 34.521 |
| libfyaml | 30.601 | 29.784 |
| rapidyaml<br>(with mutable buff) | 142.444 | 144.956 |
| rapidyaml<br>(with immutable buff) | 144.194 | 142.680 |
| yaml-cpp | 7.228 | 7.278 |
| Benchmark | Release (MB/s) |
| ---------------------------------- | -------------- |
| fkYAML | 40.491 |
| libfyaml | 31.110 |
| rapidyaml<br>(with mutable buff) | 147.221 |
| rapidyaml<br>(with immutable buff) | 144.904 |
| yaml-cpp | 7.397 |

Although [rapidyaml](https://github.com/biojppm/rapidyaml) is in general 4x faster than fkYAML as it focuses on high performance, fkYAML is 14% faster than [libfyaml](https://github.com/pantoniou/libfyaml) and also 4.8x faster than [yaml-cpp](https://github.com/jbeder/yaml-cpp).
Although [rapidyaml](https://github.com/biojppm/rapidyaml) is in general 4x faster than fkYAML as it focuses on high performance, fkYAML is 30% faster than [libfyaml](https://github.com/pantoniou/libfyaml) and also 5.4x faster than [yaml-cpp](https://github.com/jbeder/yaml-cpp).
Note that, since fkYAML deserializes scalars into native booleans or integers during the parsing, the performance could be more faster in some real use cases.

## Supported compilers
Expand Down
37 changes: 19 additions & 18 deletions include/fkYAML/detail/input/input_adapter.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ class iterator_input_adapter<
/// @return std::char_traits<char_type>::int_type A character or EOF.
void fill_buffer(std::string& buffer) {
buffer.clear();
buffer.reserve(std::distance(m_current, m_end));

switch (m_encode_type) {
case utf_encode_t::UTF_8:
Expand Down Expand Up @@ -165,7 +166,6 @@ class iterator_input_adapter<

if (consumed_size == 1) {
encoded_buffer[0] = encoded_buffer[1];
encoded_buffer[1] = 0;
}
encoded_buf_size -= consumed_size;

Expand Down Expand Up @@ -288,6 +288,7 @@ class iterator_input_adapter<
}
}

buffer.reserve(std::distance(m_current, m_end));
while (m_current != m_end) {
char c = char(*m_current++);
if (c != '\r') {
Expand Down Expand Up @@ -345,6 +346,8 @@ class iterator_input_adapter<
std::array<uint8_t, 4> utf8_buffer {{0, 0, 0, 0}};
uint32_t utf8_buf_size {0};

buffer.reserve(std::distance(m_current, m_end) * 2);

while (m_current != m_end || encoded_buf_size != 0) {
while (m_current != m_end && encoded_buf_size < 2) {
char16_t utf16 = *m_current++;
Expand Down Expand Up @@ -421,6 +424,8 @@ class iterator_input_adapter<
std::array<uint8_t, 4> utf8_buffer {{0, 0, 0, 0}};
uint32_t utf8_buf_size {0};

buffer.reserve(std::distance(m_current, m_end) * 4);

while (m_current != m_end) {
char32_t tmp = *m_current++;
char32_t utf32 = char32_t(
Expand Down Expand Up @@ -494,16 +499,17 @@ class file_input_adapter {
FK_YAML_ASSERT(m_encode_type == utf_encode_t::UTF_8);

char tmp_buf[256] {};
std::size_t buf_size = sizeof(tmp_buf) / sizeof(tmp_buf[0]);
std::size_t read_size = 0;
while ((read_size = std::fread(&tmp_buf[0], sizeof(char), sizeof(tmp_buf) / sizeof(tmp_buf[0]), m_file)) > 0) {
while ((read_size = std::fread(&tmp_buf[0], sizeof(char), buf_size, m_file)) > 0) {
char* p_current = &tmp_buf[0];
char* p_end = p_current + read_size;
do {
// find CR in `tmp_buf`.
char* p_cr_or_end = p_end;
for (uint32_t i = 0; p_current + i != p_end; i++) {
if (*(p_current + i) == '\r') {
p_cr_or_end = p_current + i;
char* p_cr_or_end = p_current;
while (p_cr_or_end != p_end) {
if (*p_cr_or_end++ == '\r') {
break;
}
}

Expand Down Expand Up @@ -560,8 +566,7 @@ class file_input_adapter {
if (m_encode_type == utf_encode_t::UTF_16BE) {
shift_bits[0] = 8;
}
else // m_encode_type == utf_encode_t::UTF_16LE
{
else { // m_encode_type == utf_encode_t::UTF_16LE
shift_bits[1] = 8;
}

Expand All @@ -586,7 +591,6 @@ class file_input_adapter {

if (consumed_size == 1) {
encoded_buffer[0] = encoded_buffer[1];
encoded_buffer[1] = 0;
}
encoded_buf_size -= consumed_size;

Expand All @@ -605,8 +609,7 @@ class file_input_adapter {
shift_bits[1] = 16;
shift_bits[2] = 8;
}
else // m_encode_type == utf_encode_t::UTF_32LE
{
else { // m_encode_type == utf_encode_t::UTF_32LE
shift_bits[1] = 8;
shift_bits[2] = 16;
shift_bits[3] = 24;
Expand Down Expand Up @@ -695,10 +698,10 @@ class stream_input_adapter {
char* p_end = p_current + read_size;
do {
// find CR in `tmp_buf`.
char* p_cr_or_end = p_end;
for (uint32_t i = 0; p_current + i != p_end; i++) {
if (*(p_current + i) == '\r') {
p_cr_or_end = p_current + i;
char* p_cr_or_end = p_current;
while (p_cr_or_end != p_end) {
if (*p_cr_or_end++ == '\r') {
break;
}
}

Expand Down Expand Up @@ -787,7 +790,6 @@ class stream_input_adapter {

if (consumed_size == 1) {
encoded_buffer[0] = encoded_buffer[1];
encoded_buffer[1] = 0;
}
encoded_buf_size -= consumed_size;

Expand All @@ -806,8 +808,7 @@ class stream_input_adapter {
shift_bits[1] = 16;
shift_bits[2] = 8;
}
else // m_encode_type == utf_encode_t::UTF_32LE
{
else { // m_encode_type == utf_encode_t::UTF_32LE
shift_bits[1] = 8;
shift_bits[2] = 16;
shift_bits[3] = 24;
Expand Down
88 changes: 50 additions & 38 deletions include/fkYAML/detail/input/lexical_analyzer.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -957,10 +957,8 @@ class lexical_analyzer {
continue;
}

uint8_t byte = static_cast<uint8_t>(current);

// Handle unescaped control characters.
if (byte <= 0x1F) {
if (static_cast<uint8_t>(current) <= 0x1F) {
handle_unescaped_control_char(current);
continue;
}
Expand Down Expand Up @@ -1032,39 +1030,40 @@ class lexical_analyzer {

uint32_t chars_in_line = 0;
bool is_extra_indented = false;
m_token_begin_itr = m_cur_itr;
if (cur_indent > indent) {
uint32_t diff = cur_indent - indent;
if (style == block_style_indicator_t::FOLDED) {
m_value_buffer.push_back('\n');
is_extra_indented = true;
}
m_value_buffer.append(diff, ' ');

uint32_t diff = cur_indent - indent;
// m_value_buffer.append(diff, ' ');
m_token_begin_itr -= diff;
chars_in_line += diff;
}

for (char current = 0; m_cur_itr != m_end_itr; ++m_cur_itr) {
current = *m_cur_itr;

if (current == '\n') {
for (; m_cur_itr != m_end_itr; ++m_cur_itr) {
if (*m_cur_itr == '\n') {
if (style == block_style_indicator_t::LITERAL) {
m_value_buffer.push_back(current);
}
else // block_style_indicator_t::FOLDED
{
if (chars_in_line == 0) {
// Just append a newline if the current line is empty.
m_value_buffer.push_back('\n');
is_extra_indented = false;
continue;
}

if (is_extra_indented) {
// A line being more indented is not folded.
m_value_buffer.push_back('\n');
chars_in_line = 0;
is_extra_indented = false;
continue;
else {
m_value_buffer.append(m_token_begin_itr, m_cur_itr + 1);
}
}
// block_style_indicator_t::FOLDED
else if (chars_in_line == 0) {
// Just append a newline if the current line is empty.
m_value_buffer.push_back('\n');
}
else if (is_extra_indented) {
// A line being more indented is not folded.
m_value_buffer.append(m_token_begin_itr, m_cur_itr + 1);
}
else {
m_value_buffer.append(m_token_begin_itr, m_cur_itr);

// Append a newline if the next line is empty.
bool is_end_of_token = false;
Expand All @@ -1075,7 +1074,7 @@ class lexical_analyzer {
break;
}

current = *m_cur_itr;
char current = *m_cur_itr;
if (current == ' ') {
continue;
}
Expand All @@ -1091,13 +1090,17 @@ class lexical_analyzer {

if (is_end_of_token) {
m_value_buffer.push_back('\n');
chars_in_line = 0;
break;
}

if (is_next_empty) {
m_value_buffer.push_back('\n');
chars_in_line = 0;
continue;
}
else {
}

switch (char next = *(m_cur_itr + 1)) {
case '\n':
Expand All @@ -1114,33 +1117,42 @@ class lexical_analyzer {
}

// Reset the values for the next line.
m_token_begin_itr = m_cur_itr + 1;
chars_in_line = 0;
is_extra_indented = false;

continue;
}

// Handle indentation
m_pos_tracker.update_position(m_cur_itr);
cur_indent = m_pos_tracker.get_cur_pos_in_line();
if (cur_indent < indent) {
if (current != ' ') {
// Interpret less indented non-space characters as the start of the next token.
break;
if (chars_in_line == 0) {
m_pos_tracker.update_position(m_cur_itr);
cur_indent = m_pos_tracker.get_cur_pos_in_line();
if (cur_indent < indent) {
if (*m_cur_itr != ' ') {
// Interpret less indented non-space characters as the start of the next token.
break;
}
// skip a space if not yet indented enough
continue;
}
// skip a space if not yet indented enough
continue;
}

if (style == block_style_indicator_t::FOLDED && chars_in_line == 0 && current == ' ') {
// A line being more indented is not folded.
m_value_buffer.push_back('\n');
is_extra_indented = true;
if (*m_cur_itr == ' ' && style == block_style_indicator_t::FOLDED) {
// A line being more indented is not folded.
m_value_buffer.push_back('\n');
is_extra_indented = true;
}
m_token_begin_itr = m_cur_itr;
}
m_value_buffer.push_back(current);

// m_value_buffer.push_back(current);
++chars_in_line;
}

if (chars_in_line > 0) {
m_value_buffer.append(m_token_begin_itr, m_cur_itr);
}

// Manipulate the trailing line endings chomping indicator type.
switch (chomp) {
case chomping_indicator_t::STRIP:
Expand Down
Loading