450kb of xhp generated HTML takes 2 seconds before jit, 1.1 seconds after (Macbook Pro, 2012)

[ejc3]

Are there any official published benchmarks of xhp? I was doing some experimentation and for relatively simple (in my opinion) HTML rendering, the performance isn't great.

For example, the following code renders get_item_simple 1000 times, generating 450kb uncompressed response and before jit, it takes 9 seconds on my mac and after it takes 1.1 seconds. Is this the expected speed of XHP? I would be happy to run tests on a official ubuntu build if you feel these numbers are anomalous.

composer:

 % cat composer.json
{
    "require": {
            "facebook/xhp-lib": "2.x"
    }
}

hhvm was built via brew and here is the version:

% hhvm --version
HipHop VM 3.15.1 (rel)
Compiler: 1476472015_N
Repo schema: 26466e8c97ddcab5e7d33c2f19029495caa6f277

here is the sample program:

<?hh

require 'vendor/autoload.php';

function get_item_simple() {
  return
  <li  >
    <div  data-test-locator="mega" >
      <div  >
        <div  ><a href="blah"  style="padding-bottom:88%;" ><img srcset="."/></a></div>
        <div  >
          <div  data-test-locator="catlabel" >blah</div>
          <div  >blah</div>
          <h3  >
            <a href="#"  >
              <u  ></u>
              <div  >blah</div>
            </a>
          </h3>
          <p  >Blah</p>
          <ul  >
            <li  >
              <a  href="." >
                <img srcset=""  alt="blah" src="blah" />
                <div  >
                  <div  >blahh</div>
                  <div  >blah</div>
                </div>
              </a>
            </li>
            <li  >
              <a  href="blah" >
                <div  >
                  <div  >blah</div>
                  <div  >blah</div>
                </div>
              </a>
            </li>
          </ul>
        </div>
      </div>
    </div>
  </li>;
}

for($i = 0; $i < 1000; $i++) {
  echo get_item_simple();
}

[ejc3]

And for reference, the raw string version is 15ms (if I switch the function to return a <<<'EOF' string instead of the xhp objects).

[fredemmott]

As part of real-world applications, XHP is usually insignificant; by itself, we wouldn't expect it to compare favorably to raw string operations, and we haven't compared it to other templating options for the last few years. It's a low enough percentage to be worth the benefits for us - though we generally don't use it for massive blobs of static content. Additionally, HHVM's Mac port is recommended for development use only; I'm not aware of any profiling or performance work happening on it.

If you're actually having a problem with this in production rather than just a microbenchmark:

• you want to set :xhp::$ENABLE_VALIDATION to false (but keep it true in development/testing)
• for massive static content, you can add a caching layer - eg https://github.com/hhvm/user-documentation/search?utf8=%E2%9C%93&q=APCCachedRenderable (performance was not an issue for actual page loads, but did get annoying when running the test suite as the suite crawls the entire site).

[ejc3]

Do you have benchmarks on the Linux version of hhvm / xhp? E.g. xhp can render N number of dom elements a second as a rough rule of thumb?

I'm trying to figure out if there is enough overhead to worry about xhp so created this microbench when I saw a larger app rendering slower than I thought. In real life, it wouldn't be static.

I'll spin a Linux box and do my test there.

[ejc3]

For completeness sake, here's the xhprof output for the test:

And here's the raw data:
xhprof.txt

[fredemmott]

No, we've not really benchmarked it recently, just investigated it when it's turned up in profiles of full applications.

If xhprof's accurate (we generally use 'perf' from linux-perf-tools, but that's not an option on OSX), the stuff at the top is already pretty minimal:

• https://github.com/facebook/xhp-lib/blob/571674608bb099c2ce01952816f606e6781dd475/src/core/ComposableElement.php#L94-L127
• https://github.com/facebook/xhp-lib/blob/571674608bb099c2ce01952816f606e6781dd475/src/core/Primitive.php#L40 is the only call to HH\Asio\m
• https://github.com/facebook/hhvm/blob/master/hphp/system/php/async/vm.ns.php#L27-L42

One thing to try is to replace a few foreach ($items as $item) { $x->add($item); } with $x->addAll($items) in replaceChildren, but that can't get you more than a 6.9% gain (almost certainly much less) based on the profile, not the orders of magnitude you're looking for.

Overall that profile looks reasonable though.

[fredemmott]

The major benefit of perf is it combines both PHP and native stacks, including kernel code; you might see something interesting in Apple's Instruments tool, though that won't be able to identify what PHP/Hack code the JITed parts of the stack refers to.

[ejc3]

Thanks for all the details and I will get this pushed to a AWS production machine to compare. I do think there's a lot of repetition in the php code, when you're trying to stamp out essentially the same template over over and over again (e.g. a stream of items, whose cdata only varies). If this actually is a problem in real life, I can see if there's any clean way to optimize that path.

[fredemmott]

It intentionally doesn't form a template from the tree, and re-using custom XHP elements is a common source of bugs; rendering the same XHP object twice is likely to be banned at some point (#135) to avoid that

[ejc3]

Okay, makes sense. Then if that was needed, two passes would make sense. Let xhp generate the template and then use a simpler engine to replace the particular pieces of cdata that need to be replaced (yes, I know that sounds ugly).

[fredemmott]

then use a simpler engine to replace the particular pieces of cdata that need to be replaced (yes, I know that sounds ugly)

This is defined by arbitrary Hack code, including awaitables, which can depend on and mutate object properties. Given the code can call setAttribute, add extra children or so on, this isn't really possible without actually turning XHP into just a template system; this would hugely limit the useful of custom elements.

That isn't impossible, but would really be something new, not XHP. It would be possible to implement that as an XHP-lib alternative, but not really as a mode for this. Implementing #64 both here and in HHVM would make that a relatively practical option.

[ejc3]

Tested on c4.2xlarge. It's about twice as fast as my MacBook Pro, 2012: 2.6 GHz Intel Core i7, both pre-jit and post-jit. On Ubuntu 16, I get 154ms after the Jit kicks in and about 900ms before.