Adding pippo benchmarks

[robertoestivill]

Adding benchmarks for Pippo

[knewmanTE]

Thanks for the pull request! Just wanted to apologize for the delay in reviewing this and merging it in; we're in the thick of preparing everything for our Round 13 release. As soon as Round 13 is over, we'll get to the Round 14 pull requests!

[knewmanTE]

Hey @robertoestivill, two things I noticed when doing some testing of Pippo:

1. Make sure you add a line in our .travis.yml file so that Pippo runs on Travis.

2. I'm getting the following errors when trying to run Pippo locally (the default "pippo" test on your FrameworkBenchmarks/pippo branch). Perhaps you'll have some insight into how to fix them:

Setup pippo: [INFO] ------------------------------------------------------------------------
Setup pippo: [INFO] Building Pippo Benchmark 0.1.0
Setup pippo: [INFO] ------------------------------------------------------------------------
Setup pippo: [WARNING] The POM for ro.pippo:pippo-core:jar:0.9.0-SNAPSHOT is missing, no dependency information available
Setup pippo: [WARNING] The POM for ro.pippo:pippo-undertow:jar:0.9.0-SNAPSHOT is missing, no dependency information available
Setup pippo: [WARNING] The POM for ro.pippo:pippo-tomcat:jar:0.9.0-SNAPSHOT is missing, no dependency information available
Setup pippo: [WARNING] The POM for ro.pippo:pippo-jetty:jar:0.9.0-SNAPSHOT is missing, no dependency information available
Setup pippo: [WARNING] The POM for ro.pippo:pippo-tjws:jar:0.9.0-SNAPSHOT is missing, no dependency information available
Setup pippo: [WARNING] The POM for ro.pippo:pippo-gson:jar:0.9.0-SNAPSHOT is missing, no dependency information available
Setup pippo: [WARNING] The POM for ro.pippo:pippo-pebble:jar:0.9.0-SNAPSHOT is missing, no dependency information available
Setup pippo: [INFO] ------------------------------------------------------------------------
Setup pippo: [INFO] BUILD FAILURE
Setup pippo: [INFO] ------------------------------------------------------------------------
Setup pippo: [INFO] Total time: 0.595 s
Setup pippo: [INFO] Finished at: 2016-09-19T15:59:37-08:00
Setup pippo: [INFO] Final Memory: 5M/15M
Setup pippo: [INFO] ------------------------------------------------------------------------
Setup pippo: [ERROR] Failed to execute goal on project pippo-benchmark: Could not resolve 
dependencies for project ro.pippo:pippo-benchmark:jar:0.1.0: The following artifacts could not be 
resolved: ro.pippo:pippo-core:jar:0.9.0-SNAPSHOT, ro.pippo:pippo-undertow:jar:0.9.0-SNAPSHOT, 
ro.pippo:pippo-tomcat:jar:0.9.0-SNAPSHOT, ro.pippo:pippo-jetty:jar:0.9.0-SNAPSHOT, ro.pippo:pippo-
tjws:jar:0.9.0-SNAPSHOT, ro.pippo:pippo-gson:jar:0.9.0-SNAPSHOT, ro.pippo:pippo-pebble:jar:0.9.0-
SNAPSHOT: Failure to find ro.pippo:pippo-core:jar:0.9.0-SNAPSHOT in 
https://oss.sonatype.org/content/repositories/snapshots was cached in the local repository, resolution 
will not be reattempted until the update interval of ossrh-snapshots has elapsed or updates are forced -
> [Help 1]
Setup pippo: [ERROR] 
Setup pippo: [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
Setup pippo: [ERROR] Re-run Maven using the -X switch to enable full debug logging.
Setup pippo: [ERROR] 
Setup pippo: [ERROR] For more information about the errors and possible solutions, please read the 
following articles:
Setup pippo: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
pippo: setup_jetty.sh and framework processes have terminated

[robertoestivill]

Yes. We were building the benchmark against a SNAPSHOT version of Pippo that included a new feature. There has been a stable release of Pippo since then, therefore I would say the old snapshot is not longer available. I will update the PR right now.

[knewmanTE]

Great, thanks @robertoestivill! One last thing: make sure to add a line to the .travis.yml file so that Travis will automatically run a verification test for your framework.

[robertoestivill]

@knewmanTE done! Thanks for the guidance :)

BTW, do all the test run periodically ? Is there any chance to see results with the current state of the PR ?

[knewmanTE]

@robertoestivill funny you should ask that. Currently, we're running all of the tests manually, and our results website also requires some manual configuration to display the results. Our general process is:

1. Run 1-2 preview rounds where we publish results from our production environments for the public to see and have an open period for people to make fixes to address issues that show up
2. Run a final round and publish that as the official "Round X Results"
3. Have an open period of a few months for people to add frameworks, tests, and larger changes to the suite
4. rinse and repeat

We're currently in step 1 for our Round 13. We ran our first preview a couple weeks ago and are finishing up a few minor things before publishing our second preview soon.

But the good news is that we're working on setting up a continuous benchmarking system. You can see the preliminary efforts in #2274. Our end goal here is to create an updated web app that is constantly updated as new continuous benchmarking results are finished.

Since your pull request is adding an entirely new framework, its official results won't be up until Round 14, but there should be a much shorter delay between round 13 and round 14 (we ran into some technical snags when switching production environments between rounds 12 and 13 that took a while to fix), and with the rate that our continuous benchmarking toolset is developing, we'll hopefully have something to show for that pretty soon too!

[knewmanTE]

@robertoestivill it looks like your postgres and mongo tests are failing.

[NateBrady23]

@robertoestivill also regarding using snapshots.. Are you removing snapshots because you're pre 1.0? We don't want to include things in the test suite that could break in a few weeks if the snaphots are being removed.

[robertoestivill]

@knewmanTE I will be taking a look early next week. I haven't go through the build logs extensively, but looks like it took way longer than any other build (32minutes)

@nbrady-techempower we will not use snapshots for the tests. It was a one time thing because a required feature was added and we needed to finish the structure of this benchmark tests. We will stick to released versions. Also, Im not a pippo maintainer, I'm just contributing to the benchmark, so I could not tell you how versioning/releases will be created in the future.

[knewmanTE]

@robertoestivill the length of the run is probably a result of a couple factors:

1. Travis is currently set up to only run a test if one of the commits on the Pull Request has changed that framework. So test runs that are 1-2 minutes are usually ones that just run an init script and then pass on that Framework
2. Compared to most frameworks, pippo is running a fairly large number of tests. Between all of the web server and database combinations, you're running 16 tests for pippo, whereas most frameworks usually run between 1-4 configuration combinations. Obviously without seeing published results it's hard to tell how much of an effect these combinations will have on the pippo results, but if, for example, it ends up looking like two of the web servers give fairly similar results, it might be worth removing one of them to simplify the results data and speed up the full suite run.

[msmith-techempower]

Seems there are errors occurring in the fortune test for postgresql and mongodb; please update.

[NateBrady23]

Hi @robertoestivill ! This looks really close. One thing to note with the new setup changes is that database dependencies now need to be included in the setup files. For instance, if a test relies on mysql, you now need to fw_depends mysql in the related setup file. I'm going to close this. If you can reopen targeting master with those changes we can go from there. Would love to get this in the next round which we will probably start doing previews for right after the new year. Thanks for the contributions! :)