Benchmark against Karma needed

[patlecat]

Hi,

I've found the speed-test below on another library page where a guy compared "format" (not a unique name btw.) against Boost Karma, among others. And Karma generally was either as fast as format or even faster.

I would love to see how you benchmark this and how you explain this. Thanks in advance.

------------------------------------snip--------------------------------------
Well, README.md of project https://github.com/vitaut/format
reference to spirit.karma, but I don't have any speed test results, so
I've cloned your project and find only one test called int-generator.
I've tested it on Visual Studio 2013 x64 compiler with full optimization (/Ox):

cl /c /W3 /WX- /Ox /Ob2 /D WIN32 /D _WINDOWS /D NDEBUG /D HAVE_BOOST /D "CMAKE_INTDIR="Release"" /D _MBCS /Gm- /EHsc /MD /GS /fp:precise /Zc:wchar_t /Zc:forScope /GR /Fo"format.dir\Release" /Fd"format.dir\Release\vc120.pdb" /Gd /TP /errorReport:prompt ..\format.cc

cl /c /W3 /WX- /Ox /Ob2 /D WIN32 /D _WINDOWS /D NDEBUG /D HAVE_BOOST /D "CMAKE_INTDIR="Release"" /D _MBCS /Gm- /EHsc /MD /GS /fp:precise /Zc:wchar_t /Zc:forScope /GR /Fo"int-generator.dir\Release" /Fd"int-generator.dir\Release\vc120.pdb" /Gd /TP /errorReport:prompt "....\format-benchmark\int_generator.cpp"

Results (MAX_ITERATION = 1e8):

ltoa: 12.3518 [s]
sprintf: 23.7584 [s]
sprintf+std::string: 25.3529 [s]
std::stringstream: 71.6845 [s]
std::to_string: 28.8839 [s]
boost::lexical_cast: 14.7594 [s]
karma::generate: 4.68553 [s]
karma::generate+std::string: 6.09963 [s]
fmt::Writer: 3.75357 [s]
fmt::Writer+std::string: 6.60581 [s]
fmt::Format: 10.4957 [s]
fmt::Format+std::string: 13.3467 [s]
fmt::FormatInt: 3.61697 [s]
cppx::decimal_from: 3.82345 [s]

As you can see boost libraries lexical_cast and karma are "among the slickest and fastest".
lexical_cast is twice faster then c-style sprintf and STL-style stringstream and to_string
and only ~20% slower then c-style ltoa.

Boost.karma already works better then fmt::* with std::string:

karma::generate+std::string: 6.09963 [s]
fmt::Writer+std::string: 6.60581 [s]
fmt::Format+std::string: 13.3467 [s]

Boost.karma works about twice faster than fmt::Format:
karma::generate: 4.68553 [s]
fmt::Format: 10.4957 [s]

FormatInt is not working with int64 type (ostringstream, to_string, boost.karma and boost.lexical_cast works fine).

So only fmt::Writer is rest:
karma::generate: 4.68553 [s]
fmt::Writer: 3.75357 [s]

Let's test it with little numbers. Patched fill function:

struct random_fill {
int operator()() const {
int scale = std::rand() / 100 + 1;

•  return ((std::rand() \* std::rand()) / scale);
  

•  return ((std::rand() \* std::rand()) / scale) % 100;
  

  }
  };

Voila!

karma::generate: 1.35417 [s]
fmt::Writer: 1.88881 [s]

And modulus 10:

struct random_fill {
int operator()() const {
int scale = std::rand() / 100 + 1;

•  return ((std::rand() \* std::rand()) / scale) % 10;
  

•  return ((std::rand() \* std::rand()) / scale);
  

  }
  };

Result (karma twice faster!):
karma::generate: 0.855458 [s]
fmt::Writer: 1.61517 [s]

[vitaut]

Thanks for your comments.

The results for the int-generator benchmark, which BTW is taken from Karma, that you give are similar to the ones I reported in http://zverovich.net/2013/09/07/integer-to-string-conversion-in-cplusplus.html .

Boost Karma is indeed pretty fast, but unfortunately it doesn't support replacement-based formatting. So it is not surprising that it is faster than fmt::Format which in addition to formatting also performs parsing of a format string and handling of positional arguments. Therefore it make more sense comparing Karma to fmt::Writer rather than to fmt::Format.

As you can see fmt::Writer is slightly faster than karma::generate on the original benchmark although it does automatic buffer management while Karma uses a fixed buffer. I'll have a look why allocating additional std::string causes larger performance impact with fmt::Writer than with Karma, but the main point is that with fmt::Writer you usually don't need this allocation, because the buffer management is automatic.

fmt::FormatInt works with int64_t now.

As for the modified benchmark, as far as I understand it compares the performance of formatting of single-digit and double-digit numbers. Since formatting of such small numbers is very fast, the buffer management that fmt::Writer does becomes a significant factor. Karma on the other hand offloads the buffer management to the user. Anyway, even for small inputs fmt::FormatInt is somewhat faster than karma::generate:

% 10 results:
karma::generate: 0.0991358 [s]
fmt::FormatInt: 0.0979277 [s]

% 100 results:
karma::generate: 0.0997507 [s]
fmt::FormatInt: 0.0943652 [s]

[vitaut]

BTW where is this library page that you've mentioned?