Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add instructions for running performance tests #4

Merged
merged 3 commits into from May 16, 2016
Merged

add instructions for running performance tests #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
24dcff1
add instructions for running performance tests
aelkiss Apr 21, 2016
ae793ab
Allow choosing thread group from command line
aelkiss May 16, 2016
34a1a1a
README for thread group command line parameters
aelkiss May 16, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
48 changes: 48 additions & 0 deletions README.md
View file
Open in desktop
Expand Up @@ -18,3 +18,51 @@ The following is an example running a test against a remote Fedora deployed unde
```bash
./jmeter -Dfedora_4_server=52.90.98.146 -Dfedora_4_context=fcrepo/rest -n -t fedora.jmx
```

### Running specific performance tests

This JMeter configuration can be used for several of the [test plans](https://wiki.duraspace.org/display/FF/Performance+and+Scalability+Test+Plans)

By default, no tests will run; tests are configured via command-line variable
definitions. To run any tests, you must set the number of threads to use to a
non-zero number via `-Dbinary_threads` or `-Dcontainer_threads`. If both are
set to nonzero, first the 'Create new containers' test will run, followed by an
attempt to run the `Create binary resource` test.

#### Test 1 - Size of files - large

The maximum filesize of 10GB as specified in the test plans does not currently seem to work for a couple reasons:
- the uploaded file is currently a single array, and Java has an array limit of between 1 and 2 billion elements
- JMeter's HTTP sampler holds the entire request in memory at once
The practical upper limit seems to be around 500MB, which still requires 6-8GB of heap for JMeter.o

Also, generating large random files is CPU-bound, so it's especially preferable
for this test to run jmeter on a separate machine from Fedora. Network
bandwidth is then likely to be the limiting factor - the underlying
RandomStringUtils implementation can generate about 100MB/second of random
data on a i7 3770.

(Ideally instead of generating the entire file at once, we could generate small
chunks at a time and just have the sampler read them from an InputStream, but
that doesn't seem to be possible with the default JMeter HTTP sampler.)

To run with one thread uploading files between 10KB and 500MB to Fedora:

* Run:
```bash
JVM_ARGS=-Xmx8G jmeter -Dfedora_4_server=<default=localhost> -Dfedora_4_context=<default=rest> -Dfilesize_min=10000 -Dfilesize_max=500000000 -Dbinary_threads=1 -n -t <path/to/fcrepo4-jmeter>/fedora.jmx
```

#### Test 2 - Size of files - small

* Run:
```bash
jmeter -Dfedora_4_server=<default=localhost> -Dfedora_4_context=<default=rest> -Dfilesize_min=0 -Dfilesize_max=4096 -Dbinary_threads=1 -n -t <path/to/fcrepo4-jmeter>/fedora.jmx
```

#### Test 4 - Number of containers - default

* Run:
```bash
jmeter -Dfedora_4_server=<default=localhost> -Dfedora_4_context=<default=rest> -Dcontainer_threads=1 -n -t <path/to/fcrepo4-jmeter>/fedora.jmx
```
105 changes: 10 additions & 95 deletions fedora.jmx
View file
Open in desktop
Expand Up @@ -12,19 +12,14 @@
<stringProp name="Argument.value">${__property(loop_count,,20)}</stringProp>
<stringProp name="Argument.metadata">=</stringProp>
</elementProp>
<elementProp name="NUM_THREADS" elementType="Argument">
<stringProp name="Argument.name">NUM_THREADS</stringProp>
<stringProp name="Argument.value">${__property(num_threads,,1)}</stringProp>
<elementProp name="CONTAINER_THREADS" elementType="Argument">
<stringProp name="Argument.name">CONTAINER_THREADS</stringProp>
<stringProp name="Argument.value">${__property(container_threads,,0)}</stringProp>
<stringProp name="Argument.metadata">=</stringProp>
</elementProp>
<elementProp name="NUM_DATASTREAMS" elementType="Argument">
<stringProp name="Argument.name">NUM_DATASTREAMS</stringProp>
<stringProp name="Argument.value">${__property(num_datastreams,,1)}</stringProp>
<stringProp name="Argument.metadata">=</stringProp>
</elementProp>
<elementProp name="NUM_DATASTREAM_MODIFICATIONS" elementType="Argument">
<stringProp name="Argument.name">NUM_DATASTREAM_MODIFICATIONS</stringProp>
<stringProp name="Argument.value">${__property(num_datastreams_modifications,, 5)}</stringProp>
<elementProp name="BINARY_THREADS" elementType="Argument">
<stringProp name="Argument.name">BINARY_THREADS</stringProp>
<stringProp name="Argument.value">${__property(binary_threads,,0)}</stringProp>
<stringProp name="Argument.metadata">=</stringProp>
</elementProp>
<elementProp name="RAMP_UP_PERIOD" elementType="Argument">
Expand Down Expand Up @@ -54,7 +49,7 @@
</elementProp>
<elementProp name="FILE_SIZE_MAX" elementType="Argument">
<stringProp name="Argument.name">FILE_SIZE_MAX</stringProp>
<stringProp name="Argument.value">${__property(filesize_mean,,100000)}</stringProp>
<stringProp name="Argument.value">${__property(filesize_max,,100000)}</stringProp>
<stringProp name="Argument.metadata">=</stringProp>
</elementProp>
<elementProp name="LOG_DIRECTORY" elementType="Argument">
Expand All @@ -77,93 +72,13 @@
<stringProp name="TestPlan.user_define_classpath"></stringProp>
</TestPlan>
<hashTree>
<ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="Fedora 4" enabled="true">
<stringProp name="ThreadGroup.on_sample_error">stoptest</stringProp>
<elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller" enabled="true">
<boolProp name="LoopController.continue_forever">false</boolProp>
<stringProp name="LoopController.loops">${LOOP_COUNT}</stringProp>
</elementProp>
<stringProp name="ThreadGroup.num_threads">${NUM_THREADS}</stringProp>
<stringProp name="ThreadGroup.ramp_time">${RAMP_UP_PERIOD}</stringProp>
<longProp name="ThreadGroup.start_time">1365510582000</longProp>
<longProp name="ThreadGroup.end_time">1365510582000</longProp>
<boolProp name="ThreadGroup.scheduler">false</boolProp>
<stringProp name="ThreadGroup.duration"></stringProp>
<stringProp name="ThreadGroup.delay"></stringProp>
</ThreadGroup>
<hashTree>
<ConfigTestElement guiclass="HttpDefaultsGui" testclass="ConfigTestElement" testname="HTTP Request Defaults" enabled="true">
<elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
<collectionProp name="Arguments.arguments"/>
</elementProp>
<stringProp name="HTTPSampler.domain">${FEDORA_4_SERVER_NAME}</stringProp>
<stringProp name="HTTPSampler.port">${FEDORA_4_SERVER_PORT}</stringProp>
<stringProp name="HTTPSampler.connect_timeout"></stringProp>
<stringProp name="HTTPSampler.response_timeout"></stringProp>
<stringProp name="HTTPSampler.protocol">http</stringProp>
<stringProp name="HTTPSampler.contentEncoding"></stringProp>
<stringProp name="HTTPSampler.path"></stringProp>
<stringProp name="HTTPSampler.implementation">HttpClient4</stringProp>
<stringProp name="HTTPSampler.concurrentPool">4</stringProp>
</ConfigTestElement>
<hashTree/>
<BSFPreProcessor guiclass="TestBeanGUI" testclass="BSFPreProcessor" testname="Set Context" enabled="true">
<stringProp name="scriptLanguage">javascript</stringProp>
<stringProp name="parameters"></stringProp>
<stringProp name="filename"></stringProp>
<stringProp name="script">vars.put(&apos;FEDORA_SERVER_CONTEXT&apos;, vars.get(&apos;FEDORA_4_SERVER_CONTEXT&apos;));
</stringProp>
</BSFPreProcessor>
<hashTree/>
<ModuleController guiclass="ModuleControllerGui" testclass="ModuleController" testname="Fedora Module" enabled="true">
<collectionProp name="ModuleController.node_path">
<stringProp name="-1227702913">WorkBench</stringProp>
<stringProp name="1667018239">Fedora Profiling</stringProp>
<stringProp name="57355947">Fedora Basic Test</stringProp>
</collectionProp>
</ModuleController>
<hashTree/>
<ResultCollector guiclass="GraphVisualizer" testclass="ResultCollector" testname="Graph Results" enabled="true">
<boolProp name="ResultCollector.error_logging">false</boolProp>
<objProp>
<name>saveConfig</name>
<value class="SampleSaveConfiguration">
<time>true</time>
<latency>true</latency>
<timestamp>true</timestamp>
<success>true</success>
<label>true</label>
<code>true</code>
<message>true</message>
<threadName>true</threadName>
<dataType>true</dataType>
<encoding>false</encoding>
<assertions>false</assertions>
<subresults>false</subresults>
<responseData>false</responseData>
<samplerData>false</samplerData>
<xml>false</xml>
<fieldNames>true</fieldNames>
<responseHeaders>false</responseHeaders>
<requestHeaders>false</requestHeaders>
<responseDataOnError>false</responseDataOnError>
<saveAssertionResultsFailureMessage>false</saveAssertionResultsFailureMessage>
<assertionsResultsToSave>0</assertionsResultsToSave>
<bytes>true</bytes>
<threadCounts>true</threadCounts>
</value>
</objProp>
<stringProp name="filename">${LOG_DIRECTORY}/jmeter-fedora4-${FILE_SIZE_MEAN}-mean-${FILE_SIZE_STD_DEV}-stddev-${NUM_THREADS}-threads.csv</stringProp>
</ResultCollector>
<hashTree/>
</hashTree>
<ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="Fedora4 Create New Containers" enabled="true">
<stringProp name="ThreadGroup.on_sample_error">stoptest</stringProp>
<elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller" enabled="true">
<boolProp name="LoopController.continue_forever">false</boolProp>
<intProp name="LoopController.loops">-1</intProp>
</elementProp>
<stringProp name="ThreadGroup.num_threads">${NUM_THREADS}</stringProp>
<stringProp name="ThreadGroup.num_threads">${CONTAINER_THREADS}</stringProp>
<stringProp name="ThreadGroup.ramp_time">${RAMP_UP_PERIOD}</stringProp>
<longProp name="ThreadGroup.start_time">1370357053000</longProp>
<longProp name="ThreadGroup.end_time">1370357053000</longProp>
Expand Down Expand Up @@ -279,7 +194,7 @@ if (elapsed &gt; threshold) {
<boolProp name="LoopController.continue_forever">false</boolProp>
<intProp name="LoopController.loops">-1</intProp>
</elementProp>
<stringProp name="ThreadGroup.num_threads">${NUM_THREADS}</stringProp>
<stringProp name="ThreadGroup.num_threads">${BINARY_THREADS}</stringProp>
<stringProp name="ThreadGroup.ramp_time">${RAMP_UP_PERIOD}</stringProp>
<longProp name="ThreadGroup.start_time">1370357053000</longProp>
<longProp name="ThreadGroup.end_time">1370357053000</longProp>
Expand Down Expand Up @@ -496,7 +411,7 @@ vars.put(&quot;post_body&quot;, RandomStringUtils.randomAscii(size));
<stringProp name="filename">/tmp/jmeter.out</stringProp>
</ResultCollector>
<hashTree/>
<TestFragmentController guiclass="TestFragmentControllerGui" testclass="TestFragmentController" testname="Performance requests" enabled="false"/>
<TestFragmentController guiclass="TestFragmentControllerGui" testclass="TestFragmentController" testname="Performance requests" enabled="true"/>
<hashTree>
<RandomVariableConfig guiclass="TestBeanGUI" testclass="RandomVariableConfig" testname="Random Variable" enabled="true">
<stringProp name="maximumValue">2147483640</stringProp>
Expand Down