Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

jmeter-maven-plugin v2.2.0 not longer processes <propertiesUser> as v1.10.1 #230

Open
itzed-solutions opened this Issue Jun 7, 2017 · 10 comments

Comments

3 participants
@itzed-solutions
Copy link

itzed-solutions commented Jun 7, 2017

I have been happily using v1.10.0 of the plugin within a RedHat environment. Recently we changed to an Ubuntu server and v1.10.0 failed to compile due to a dependencies issue. However, it is a few years old, so I decided to update to the most recent version. However this exhibits a different issue regarding the handling of the propertiesUser element.

Within v1.10.0 I use the following structure,

		<plugin>
            <groupId>com.lazerycode.jmeter</groupId>
            <artifactId>jmeter-maven-plugin</artifactId>
            <version>1.10.1</version>
			<dependencies>
				<!-- https://mvnrepository.com/artifact/mysql/mysql-connector-java -->
				<dependency>
				  <groupId>mysql</groupId>
				  <artifactId>mysql-connector-java</artifactId>
				  <version>5.1.39</version>
				</dependency>
			</dependencies>
		  <configuration>
			<!--
			   By default the test results are saved in a file  
			   /target/jmeter/results/<testname>-<timestamp>.jtl
			   Further processing is easier without timestamp though.
			-->
			<testResultsTimestamp>false</testResultsTimestamp>
		 
			<!--
			   To simplify debugging, it is advisable to adapt the loglevel.
			   The jmeter logs are written to the file jmeter.log.
			-->
			<overrideRootLogLevel>DEBUG</overrideRootLogLevel>
		 
			<!--
			   By default, the console output during a jmeter test run is suppressed.
			   We want to display the progress using the listener "Generate Summary Results" 
			   (which periodically prints stats to stdout). Therefore we have to make sure,
			   that the jmeter output is not suppressed.
			-->
			<suppressJMeterOutput>false</suppressJMeterOutput>
		 
			<!--
			   If tests fail (e.g. a http-request running into a timeout), the corresponding maven
			   goal also fails (and subsequent goals aren't executed anymore). We want to create graphs
			   from test-results, no matter if some requests failed or not, so we ignore jmeter failures.
			-->
			<ignoreResultFailures>false</ignoreResultFailures>
		  </configuration>
		  <executions>
                <execution>
                    <id>JMETER-GetNumbers</id>
                    <phase>verify</phase>
                    <goals>
						<goal>jmeter</goal>
                    </goals>
					<configuration>
						<testFilesIncluded>
							<jMeterTestFile>JmeterGetNumbers.jmx</jMeterTestFile>
						</testFilesIncluded>
						<propertiesUser>
							<users>1</users>
							<iterations>1</iterations>
							<rampup>0.1</rampup>
							<testsuite>123</testsuite>
							<testcase>test</testcase>
							<configfile>../../../configuration/TestSuite.xls</configfile>
							<testscriptpath>../../../configuration</testscriptpath>
							<server>server1</server>
							<port>1234</port>
							<timeout>8000</timeout>
						</propertiesUser>
					</configuration>
                </execution>
                <execution>
                    <id>TestSuite</id>
                    <phase>verify</phase>
                    <goals>
                        <goal>jmeter</goal>
                    </goals>
					<configuration>
						<testFilesIncluded>
							<jMeterTestFile>JmeterTestSuite.jmx</jMeterTestFile>
						</testFilesIncluded>
						<!--resultsFileNameDateFormat>YYYMMDD-HHmmss</resultsFileNameDateFormat>
                        <testResultsTimestamp>true</testResultsTimestamp-->
						<propertiesUser>
							<users>1</users>
							<iterations>96</iterations>
							<rampup>2</rampup>
							<testsuite>123</testsuite>
							<testcase>123</testcase>
							<server>server2</server>
							<port>1234</port>
							<path>path1</path>
							<configfile>../../../configuration/TestSuite.xls</configfile>
							<testscriptpath>../../../configuration</testscriptpath>
							<timeout>6000</timeout>
						</propertiesUser>
					</configuration>
                </execution>
			</executions>
        </plugin>

The above has been working for many months. However when I upgrade to v2.2.0, the plugin seems to ignore the propertiesUser when specified within the . The log shows a failure to locate the test configuration due to testscriptpath using the default value specified in the .jmx file ... ${_P(testscriptpath,e:/business/xxxxxx/testing/jmeter/xxxxxx/xxxxxxxx)}/xxxxxxxx${__threadNum}.csv. As you can see the default is a windows path and hence not going to work on Ubuntu. The log shows that it is trying to execute two different *.jmx files, so the testFilesIncluded within the block within the is working.

I have moved the propertiesUser to the block external to the executions block and this is then accepted, but of course then applies to all execution which is not what I want, or what previously worked.

I'm sure I can work around this, by only using a single execution per POM.xml. But this is a lot of work and I wanted to check if this is a known issue and whether there may be a workaround or fix for this.

Many thanks

Ed

@Ardesco

This comment has been minimized.

Copy link
Contributor

Ardesco commented Jun 7, 2017

Can you provide the actual plugins POM entry you are using with 2.2.0 (or are you using the above but just changing the version number?)

version 2.x.x brought in a series of breaking changes. We now run different parts of the plugin in different maven phases, so configuration changes need to be linked to the correct phase, or generic configuration changes that apply to all phases. We also changed the way we pull down dependencies to make things easier to manage.

The best place to start is the CHANGELOG: https://github.com/jmeter-maven-plugin/jmeter-maven-plugin/blob/master/CHANGELOG.md

@itzed-solutions

This comment has been minimized.

Copy link
Author

itzed-solutions commented Jun 7, 2017

Hi, the POM is as above , but just changing the version number. I noticed also that the editor remove some of the text I added to the original post ... so I have updated this to make it make more sense.

I've taken a look at the CHANGELOG, but nothing really jumps out at me. Which of the changes do you think has affected the propertiesUser in this way?

@Ardesco

This comment has been minimized.

Copy link
Contributor

Ardesco commented Jun 7, 2017

You are setting the propertiesUser configuration in the verify phase, it needs to be set in the compile phase. You have 2 options

  1. Pull the configuration block out of the execution block and set it under the main plugin configuration block.
  2. Move your configuration into new execution block for the compile phase and the configure goal.

I expect you to then have problems with your mysql-connector-java dependency, have a look at this to fix that problem:

https://github.com/jmeter-maven-plugin/jmeter-maven-plugin/wiki/Adding-additional-libraries-to-the-classpath

@itzed-solutions

This comment has been minimized.

Copy link
Author

itzed-solutions commented Jun 7, 2017

Thank you for your rapid response. If I go for option 2, let me make sure I understand. My current POM has the structure:

executions
execution#1-phase:verify-goal:jmeter-propertiesUser#1
execution#2-phase:verify-goal:jmeter-propertiesUser#2
executions

With option2 the structure would be ;
executions
execution#1-phase:compile-goal:configure-propertiesUser#1
execution#2-phase:verify-goal:jmeter
execution#3-phase:compile-goal:configure-propertiesUser#2
execution#4-phase:verify-goal:jmeter
executions

Is this right?

Regarding the dependencies, yes I noticed that this had changed and have updated it.

Ed

@Ardesco

This comment has been minimized.

Copy link
Contributor

Ardesco commented Jun 7, 2017

Probably useful for you to have a read of this:

https://maven.apache.org/guides/mini/guide-configuring-plugins.html#Configuring_Build_Plugins

I would expect your execution block to look like this:

<executions>
    <execution>
        <phase>compile</phase>
        <goals>
            <goal>configure</goal>
        </goals>
        <configuration>
            <testFilesIncluded>
                <jMeterTestFile>JmeterTestSuite.jmx</jMeterTestFile>
            </testFilesIncluded>
            <propertiesUser>
                <users>1</users>
                <iterations>96</iterations>
                <rampup>2</rampup>
                <testsuite>123</testsuite>
                <testcase>123</testcase>
                <server>server2</server>
                <port>1234</port>
                <path>path1</path>
                <configfile>../../../configuration/TestSuite.xls</configfile>
                <testscriptpath>../../../configuration</testscriptpath>
                <timeout>6000</timeout>
            </propertiesUser>
        </configuration>
    </execution>
    <execution>
        <phase>verify</phase>
        <goals>
            <goal>jmeter</goal>
        </goals>
    </execution>
</executions>
@itzed-solutions

This comment has been minimized.

Copy link
Author

itzed-solutions commented Jun 7, 2017

OK. I'm making some progress. Thank you for your help so far Mark. I followed your example above and things seems to start working. Just a few further questions

[1] Jmeter seems to execute all of the jmx files in src/test/jmeter rather than just the ones I have in the POM?
[2] I'm using a SOAPUI MOCK in conjuction with JMETER to perform some black box testing. If I make the changes you suggest to the jmeter plugin the soapui plugin stops working and looks for the .xml SOAPUI config file in the target/jmeter/bin? Here is the snippet from the POM.
[3] When I execute the two jmx tests as in the POM ... the order seems to be switched in execution. How is the order controlled?
[4] The test that uses mysql fails ... the results file states
sample t="2" it="0" lt="0" ct="2" ts="1496845552283" s="false" lb="AMPDatabaseQuery-poiTransactionNumber" rc="null 0" rm="java.sql.SQLException: Cannot load JDBC driver class 'com.mysql.jdbc.Driver'" tn="SyncPoiTransactionNumberWithAMP 1-1" dt="text" by="53" sby="0" ng="1" na="1". I think i have the jmeterExtensions in the right place?

POM.xml snippet

    <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-surefire-report-plugin</artifactId>
                            <version>2.19.1</version>
                </plugin>
                <plugin>
                            <groupId>com.smartbear.soapui</groupId>
                            <artifactId>soapui-maven-plugin</artifactId>
                            <version>5.2.1</version>
                            <!--groupId>com.github.redfish4ktc.soapui</groupId>
                            <artifactId>maven-soapui-extension-plugin</artifactId>
                            <version>4.6.4.2</version-->
            <executions>
                <execution>
                    <phase>test</phase>
                    <goals>
                        <goal>mock</goal>
                    </goals>
                    <configuration>
                        <projectFile>IntralotATG2.0LoadTesting.xml</projectFile>
                                                    <outputFolder>${project.build.directory}/surefire-reports</outputFolder>
                                                    <junitReport>true</junitReport>
                                                    <exportAll>true</exportAll>
                                                    <printReport>true</printReport>
                                                    <testSuite>Authenticate</testSuite>
                                                    <port>8280</port>
                                                    <path>/DLInlineSalesBean</path>
                                                    <mockService>MockService</mockService>
                                                    <noBlock>true</noBlock>
                        <projectProperties>
                                                            <projectProperty>testScriptPath=configuration/SOAPUI/001/</projectProperty>
                        </projectProperties>
                    </configuration>
                </execution>
            </executions>
                    </plugin>
                    <plugin>
            <groupId>com.lazerycode.jmeter</groupId>
            <artifactId>jmeter-maven-plugin</artifactId>
            <!--version>1.10.1</version-->
            <version>2.2.0</version>
                            <!--dependencies>
                                    <dependency>
                                      <groupId>mysql</groupId>
                                      <artifactId>mysql-connector-java</artifactId>
                                      <version>5.1.39</version>
                                    </dependency>
                            </dependencies-->
                            <configuration>
                                    <!--
                                       By default the test results are saved in a file
                                       /target/jmeter/results/<testname>-<timestamp>.jtl
                                       Further processing is easier without timestamp though.
                                    -->
                                    <testResultsTimestamp>false</testResultsTimestamp>

                                    <!--
                                       To simplify debugging, it is advisable to adapt the loglevel.
                                       The jmeter logs are written to the file jmeter.log.
                                    -->
                                    <overrideRootLogLevel>DEBUG</overrideRootLogLevel>

                                    <!--
                                       By default, the console output during a jmeter test run is suppressed.
                                       We want to display the progress using the listener "Generate Summary Results"
                                       (which periodically prints stats to stdout). Therefore we have to make sure,
                                       that the jmeter output is not suppressed.
                                    -->
                                    <suppressJMeterOutput>false</suppressJMeterOutput>

                                    <!--
                                       If tests fail (e.g. a http-request running into a timeout), the corresponding maven
                                       goal also fails (and subsequent goals aren't executed anymore). We want to create graphs
                                       from test-results, no matter if some requests failed or not, so we ignore jmeter failures.
                                    -->
                                    <ignoreResultFailures>true</ignoreResultFailures>
                            <jmeterExtensions>
                                      <artifact>mysql:mysql-connector-java:pom:5.1.39</artifact>
                            </jmeterExtensions>
                            </configuration>
                            <executions>
                <execution>
                    <id>SET CONFIG FOR PHASE1</id>
                    <phase>compile</phase>
                    <goals>
                                                    <goal>configure</goal>
                    </goals>
                            <configuration>
                                                    <testFilesIncluded>
                                                            <jMeterTestFile>JmeterLoadTestGetATGPoiTransactionNumbers.jmx</jMeterTestFile>
                                                    </testFilesIncluded>
                                                    <propertiesUser>
                                                            <users>1</users>
                                                            <iterations>1</iterations>
                                                            <rampup>0.1</rampup>
                                                            <testscriptpath>../../../configuration/JMETER</testscriptpath>
                                                            <server>asi.amp.com</server>
                                                            <port>3306</port>
                                                            <timeout>8000</timeout>
                                                    </propertiesUser>
                            </configuration>
                </execution>
                <execution>
                    <id>SET CONFIG FOR PHASE2</id>
                    <phase>compile</phase>
                    <goals>
                                                    <goal>configure</goal>
                    </goals>
                            <configuration>
                                                    <testFilesIncluded>
                                                            <jMeterTestFile>JmeterLoadTest5155Rdm.jmx</jMeterTestFile>
                                                    </testFilesIncluded>
                                                    <propertiesUser>
                                                            <users>1</users>
                                                            <iterations>1</iterations>
                                                            <rampup>0.1</rampup>
                                                            <testscriptpath>../../../configuration/JMETER</testscriptpath>
                                                            <server>asi.com</server>
                                                            <port>9099</port>
                                                            <timeout>8000</timeout>
                                                    </propertiesUser>
                            </configuration>
                </execution>
                <execution>
                    <id>JMETER-GetATGPoiTransactionNumbers-JmeterLoadTest5155Rdm</id>
                    <phase>verify</phase>
                    <goals>
                                                    <goal>jmeter</goal>
                    </goals>
                                            <!--configuration>
                                                    <propertiesUser>
                                                            <users>1</users>
                                                            <iterations>1</iterations>
                                                            <rampup>0.1</rampup>
                                                            <testscriptpath>../../../configuration/JMETER</testscriptpath>
                                                            <server>asi.amp.com</server>
                                                            <port>3306</port>
                                                            <timeout>8000</timeout>
                                                    </propertiesUser>
                                                    <testFilesIncluded>
                                                            <jMeterTestFile>JmeterLoadTestGetATGPoiTransactionNumbers.jmx</jMeterTestFile>
                                                    </testFilesIncluded>
                                            </configuration-->
                </execution>
            </executions>
        </plugin>
    </plugins>
@itzed-solutions

This comment has been minimized.

Copy link
Author

itzed-solutions commented Jun 7, 2017

[1]&[3] Resolved .. moved the testFilesIncluded back into the verify phase and added an execution for each test
[3] Resolved - I don't understand why this is happening as they are different plugins. But I have resolved it by adding an absolute path to the SOAPUI xml file
[4] Resolved ... mysql:mysql-connector-java:pom:5.1.39 should have been mysql:mysql-connector-java:5.1.39. My mistake.

[5] However the main issue still exists that I wish to run 2 tests with the same set of propertiesUser, BUT with different values for each test... At the start of the test I add 2 configurations. However one overwrites the other, therefore both tests use the same values. Is there some way to link a specific compile step to a specific verify phase?

POM now looks like this:

                    <plugin>
            <groupId>com.lazerycode.jmeter</groupId>
            <artifactId>jmeter-maven-plugin</artifactId>
            <!--version>1.10.1</version-->
            <version>2.2.0</version>
                            <configuration>
                                    <!--
                                       By default the test results are saved in a file
                                       /target/jmeter/results/<testname>-<timestamp>.jtl
                                       Further processing is easier without timestamp though.
                                    -->
                                    <testResultsTimestamp>false</testResultsTimestamp>

                                    <!--
                                       To simplify debugging, it is advisable to adapt the loglevel.
                                       The jmeter logs are written to the file jmeter.log.
                                    -->
                                    <overrideRootLogLevel>DEBUG</overrideRootLogLevel>

                                    <!--
                                       By default, the console output during a jmeter test run is suppressed.
                                       We want to display the progress using the listener "Generate Summary Results"
                                       (which periodically prints stats to stdout). Therefore we have to make sure,
                                       that the jmeter output is not suppressed.
                                    -->
                                    <suppressJMeterOutput>false</suppressJMeterOutput>

                                    <!--
                                       If tests fail (e.g. a http-request running into a timeout), the corresponding maven
                                       goal also fails (and subsequent goals aren't executed anymore). We want to create graphs
                                       from test-results, no matter if some requests failed or not, so we ignore jmeter failures.
                                    -->
                                    <ignoreResultFailures>true</ignoreResultFailures>
                            <jmeterExtensions>
                                      <artifact>mysql:mysql-connector-java:5.1.39</artifact>
                            </jmeterExtensions>
                            </configuration>
                            <executions>
                <execution>
                    <id>SET CONFIG FOR PHASE1</id>
                    <phase>compile</phase>
                    <goals>
                                                    <goal>configure</goal>
                    </goals>
                            <configuration>
                                                    <propertiesUser>
                                                            <users>1</users>
                                                            <iterations>1</iterations>
                                                            <rampup>0.1</rampup>
                                                            <testscriptpath>../../../configuration/JMETER</testscriptpath>
                                                            <server>asi.amp.com</server>
                                                            <port>3306</port>
                                                            <timeout>8000</timeout>
                                                    </propertiesUser>
                            </configuration>
                </execution>
                <execution>
                    <id>SET CONFIG FOR PHASE2</id>
                    <phase>compile</phase>
                    <goals>
                                                    <goal>configure</goal>
                    </goals>
                            <configuration>
                                                    <propertiesUser>
                                                            <users>1</users>
                                                            <iterations>1</iterations>
                                                            <rampup>0.1</rampup>
                                                            <testscriptpath>../../../configuration/JMETER</testscriptpath>
                                                            <server>asi.com</server>
                                                            <port>9099</port>
                                                            <timeout>8000</timeout>
                                                    </propertiesUser>
                            </configuration>
                </execution>
                <execution>
                    <id>JMETER-GetATGPoiTransactionNumbers-JmeterLoadTestGetATGPoiTransactionNumbers</id>
                    <phase>verify</phase>
                    <goals>
                                                    <goal>jmeter</goal>
                    </goals>
                            <configuration>
                                                    <testFilesIncluded>
                                                            <jMeterTestFile>JmeterLoadTestGetATGPoiTransactionNumbers.jmx</jMeterTestFile>
                                                    </testFilesIncluded>
                            </configuration>
                </execution>
                <execution>
                    <id>JMETER-GetATGPoiTransactionNumbers-JmeterLoadTest5155Rdm</id>
                    <phase>verify</phase>
                    <goals>
                                                    <goal>jmeter</goal>
                    </goals>
                            <configuration>
                                                    <testFilesIncluded>
                                                            <jMeterTestFile>JmeterLoadTest5155Rdm.jmx</jMeterTestFile>
                                                    </testFilesIncluded>
                            </configuration>
                </execution>
            </executions>
        </plugin>
@Ardesco

This comment has been minimized.

Copy link
Contributor

Ardesco commented Jun 7, 2017

Took me a while but I understand the problem that you are seeing now.

The plugin currently uses 3 maven phases.

  1. COMPILE - To copy files across into the target directory and set JMeter up to run tests
  2. INTEGRATION_TEST - To actually run tests
  3. VERIFY - To scan results created by the tests

Maven phases run in a strict order, so all actions that take place in the compile phase run first, then all actions that occur in the integration test phase will happen afterwards (See https://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html for more information on maven phases).

This results in your compile phases being executed one after the other before maven gets to the integration tests phase. This obviously means that your second compile phase is overwriting your first compile phase.

The main problem is that we copy everything into a src/test/jmeter directory in the target directory. To try and work around this problem we would need to change this to something like <execution_id>/src/test/jmeter. However I'm not sure if we can pull the execution id of an execution block out yet. This will need some more investigation.

@Ardesco

This comment has been minimized.

Copy link
Contributor

Ardesco commented Jun 7, 2017

Right, instead of using a config setting of:

	/**
	 * Place where the JMeter files will be generated.
	 */
	@Parameter(defaultValue = "${project.build.directory}/jmeter")
	protected transient File jmeterDirectory;

We are going to have to generate this on the fly based upon the execution id, something like:

projectBuildDirectory + File.separator + this.mojoExecution.getExecutionId() + File.separator + "jmeter"

This means that configuration is going to have to write the various directories used to file and running the tests is going to have to read these directories from file. We currently write stuff to a config.json to keep track of results file locations, we should probably add source file locations to this file.

A lot of the above is notes for when we implement this :)

@Ardesco Ardesco added bugs and removed investigation labels Jun 7, 2017

@Ardesco Ardesco self-assigned this Jun 7, 2017

@itzed-solutions

This comment has been minimized.

Copy link
Author

itzed-solutions commented Jun 8, 2017

Mark, If I understand the above correctly. The functionality available in v1.10.0 is no longer supported by 2.2.0 without a fix. So in the mean time, before the fix is available I will have to split the current POM (2xcompile,2xverify phase) into 2 separate POMs (1xcompile,1xverify phase) and execute these from 2 separate jenkins projects and link these together within jenkins. I need to get it working this week, so I would think the workaround is the only option. Do you know how long a fix might take? I have many(!) POM of this type and it is a fair amount of work to change them all.

If you need me to do any testing of the fix etc, then just let me know. I'd be more than happy to help.

Many thanks for your help with this issue

Ed

@pmouawad pmouawad added the help wanted label Feb 1, 2018

@pmouawad pmouawad added this to the 2.8.0 milestone Feb 11, 2018

@pmouawad pmouawad added this to Low priority in ISSUE TRACKING Aug 3, 2018

@pmouawad pmouawad modified the milestones: 2.8.0, 2.9.0 Oct 4, 2018

@pmouawad pmouawad modified the milestones: 2.8.1, 2.8.2 Nov 11, 2018

@pmouawad pmouawad modified the milestones: 2.8.2, 2.8.3 Dec 27, 2018

@pmouawad pmouawad modified the milestones: 2.8.3, 2.8.4 Jan 15, 2019

@pmouawad pmouawad modified the milestones: 2.8.4, 2.8.5 Jan 29, 2019

@pmouawad pmouawad modified the milestones: 2.8.6, 2.8.7 Feb 19, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.