Proposal for next test type: cached multiple database queries

[bhauer]

I'd like to propose and discuss the next test type now so that we can flesh out the details and allow implementations to begin soon. I'm leaning toward the next type being a caching test. To keep things simple, I propose basing the test on the existing multiple database query test.

The requirements at a high level would be:

1. From the client's perspective, the test is identical in response behavior to the existing multiple query test. The client can provide a URL parameter specifying the number of random World objects it wants, and the server furnishes a random list of that size in response, serialized as JSON.
2. The server may use any caching library or approach desired, except for reverse-proxy caching. This project is expressly not a test of reverse-proxy caches (Varnish, nginx caching, Apache's mod_cache, and so on). In any event, the randomness requirement of the existing multiple-query test would make reverse-proxying the response invalid. That said, the implementation may use an in-process memory cache, a separate-process memory cache, or any other conceivable approach (within reasonable limits, of course).
3. The implementation must not require an additional server instance or additional hardware. The cache should run alongside the application server (either in-process or separate process on localhost).
4. The cache server must be startable and stoppable in the same fashion as the application servers. For isolation, even where two implementations use the same cache technology (e.g., if two implementations were to use Memcached), the two instances of Memcached should be installed and managed independently. I'd prefer to not manage and clean up cache server instances that are shared across test implementations.
5. It might go without saying, but the application server must communicate with the cache; the implementation should not expect that the client communicate with the cache directly.
6. As with other test types, a warm-up run will occur prior to capturing performance data. However, it is also acceptable for the implementations' setup scripts to prime the cache in whatever form needed, assuming that priming process completes in a reasonable amount of time.
7. The implementation should use a bona-fide cache library or server. This is admittedly a subjective matter, but we want to avoid implementations that just create a self-managed plain old map of objects. Admittedly, there will be some gray area. We want for the implementation to be as realistic as possible, so please use a library that is realistic for a production environment. To set a bare minimum, for those familiar with Google's Guava library, Guava's CacheBuilder would be the simplest implementation we'd like to consider.
8. Use of bulk operations is acceptable. If the cache library allows for the fetching of several World objects identified by random identities in a single cache operation, implementations are welcome to do so.
9. This test type is not exercising a distributed cache (that would be a separate test type), so it is not necessary to set up a multi-node configuration. See above. To reiterate, for deployment simplicity, if an implementation uses a separate-process cache, we want to have the cache server deployed on the same hardware/instance as the application server.

[bhauer]

Added 7, 8, and 9 to the above description.

We expect a bona-fide cache library; bulk get operations are acceptable; and there is no need to configure a distributed cache.

[bhauer]

While the multi-query test runs 1, 5, 10, and 20 iterations, we should consider having this test using more iterations. Perhaps 1, 20, 100, 500.

[bhauer]

What do you guys think about starting to tackle reference implementations of this? It should be really simple to add a Guava CacheBuilder implementation to Servlet, create a Gemini version (just create a different entity class that is marked as cached but refers to the same Worlds table). Then with a bit more work we could create Rails, Django, and node.js tests that use Redis, for example. There may even be libraries for those that fairly easily add a pass-through Redis cache.

At the very least, I'd like to finalize the specifications and allow implementations to begin. Are there any concerns to raise about the specifications enumerated above? Especially of interest to me are any loopholes that exotic implementations may exploit.

[msmith-techempower]

Kicking this WAY down the road since this will not be happening in round 12.

[zane-techempower]

1. @bhauer should this test be realistic in that the caching layer should be in front of the database? Or can tests implement this as "look for value in cache, otherwise not found"?

2. Let's try to put together a list of frameworks that we want to have done for-sure for round 14. We talked about doing some Java frameworks, then perhaps some Go and Javascript ones to get us at least 10-20 implementations of this test for the first round that it appears in. Here is a start:
   gemini-mysql, gemini-postgres, servlet-raw, undertow/undertow-edge (there seem to be multiple implementations of each of these), dropwizard, grizzly-jersey, go-prefork, go-raw, go, gin, goji, nodejs, nodejs-mysql, express-mysql

[bhauer]

Sorry, I accidentally commented on this while using a different profile.

@zane-techempower 1) The implementation should be a cache backed by the database table. It is acceptable for the cache to be primed with all values from the table at start-up time or to populate lazily on an as-needed basis. But it should not return "not found" for records that do exist in the database table. Note that there is no expectation of concurrent updates to the table, so it's not necessary to periodically examine the table for changes.

2. If possible, I'd also like to see one framework each representing Python, Ruby, and PHP.

[greenlaw110]

@zane-techempower I think there should be some standard way in terms of setting up the cache service provider. We don't want measure the differences between IPC and in-process communication, do we? Let's say if we decide to use memcached then every framework/platform test bed shall use memcached. It is unfair to compare memcached based cache vs hash table based cache built by Guava's CacheBuilder.

[bhauer]

@greenlaw110 While I appreciate that point of view, as we design and specify test types such as this one, I feel we need to accept and allow for the wide spectrum of best-practices that may exist throughout all of the languages, platforms and frameworks.

The more functionality we expect in a test type, the more diversity we need to be prepared to expect in implementation approaches. We can set some constraints if we feel it's warranted, but for this test type, I think the constraints I've outlined above are fairly reasonable. Perhaps we can add a distributed cache test as a future test type and require use of, for example, Redis. But even there, my inclination would be to allow for any distributed cache according to the best practices or preference of the frameworks.

A test like this one allows platforms that have robust in-VM caching capabilities to shine. Requiring use of an external service, when several platform have in-VM capabilities that meet our requirements, makes this test type too similar to the database tests (that is, all implementations must make requests to an external service) for my liking.

[NateBrady23]

This new test has been merged into master with specs here: http://frameworkbenchmarks.readthedocs.io/en/latest/Project-Information/Framework-Tests/

[bhauer]

For anyone interested, initial data for this new test type is starting to show up in the continuous results. For example, view http://simshill.techempower.com/raw/results.2018-02-09-10-22-34-441.json and navigate to:

rawData > cached_query

We likely won't publish data for this test type on the results web site until we have a few more implementations. It would be nice to have 20+ but even a dozen or so would be a good target. Some additional diversity in technology coverage would be nice too.