Skip to content

Fix InfluxDBRawBackendListenerClient missing data #5785

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

vperaltac
Copy link
Contributor

@vperaltac vperaltac commented Mar 6, 2023

Description

This PR includes a new tag threadName for InfluxDBRawBackendListenerClient. This allows InfluxDB to insert multiple entries with the same timestamp but with different threadName.

Motivation and Context

fixes #5654

There is currently an issue where Jmeter will send requests coming from multiple threads where the timestamp for those requests is the exact same, this causes InfluxDB to override that data and only keep 1 entry.
Here is an example with a test that runs 12 requests:

time                connectTime duration status transaction  ttfb
----                ----------- -------- ------ -----------  ----
1678045696972000000 56          72       ok     user2        72
1678045698040000000 0           16       ok     userlist     16
1678045698044000000 0           25       ok     userlist     25
1678045699056000000 0           19       ok     listresource 19
1678045699070000000 0           15       ok     listresource 14
1678045700076000000 30          46       ok     user2        46
1678045700085000000 34          50       ok     user2        50
1678045701122000000 0           14       ok     userlist     14
1678045701136000000 0           15       ok     userlist     15
1678045702136000000 0           16       ok     listresource 16
1678045702150000000 0           16       ok     listresource 16

In this example there is 1 entry missing at the beginning, there should be 2 entries for the transaction user2. The reason why the second entry is not there is because both had the same timestamp, status and transaction values.

After adding the tag threadName, InfluxDB will save both entries since they differ in the value for that tag. Notice how the time for the 2 first entries is the same:

time                connectTime duration status threadName     transaction  ttfb
----                ----------- -------- ------ ----------     -----------  ----
1678046919362000000 39          55       ok     ThreadGroup1-1 user2        55
1678046919362000000 36          59       ok     ThreadGroup1-2 user2        59
1678046920418000000 0           19       ok     ThreadGroup1-1 userlist     19
1678046920421000000 0           18       ok     ThreadGroup1-2 userlist     18
1678046921438000000 0           18       ok     ThreadGroup1-1 listresource 18
1678046921439000000 0           21       ok     ThreadGroup1-2 listresource 21
1678046922457000000 38          53       ok     ThreadGroup1-1 user2        53
1678046922461000000 37          56       ok     ThreadGroup1-2 user2        56
1678046923511000000 0           17       ok     ThreadGroup1-1 userlist     17
1678046923517000000 0           18       ok     ThreadGroup1-2 userlist     18
1678046924528000000 0           26       ok     ThreadGroup1-1 listresource 26
1678046924536000000 0           16       ok     ThreadGroup1-2 listresource 16

This change removes any white spaces in the thread name, but given that this value can be set by the user, maybe this is not enough, I would appreciate some feedback here.

How Has This Been Tested?

Ran multiple tests after making the changes with over 3000 requests, did not lose any data this time.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)

Checklist:

  • My code follows the code style of this project.
  • I have updated the documentation accordingly.

@ham1
Copy link
Contributor

ham1 commented Mar 10, 2023

LGTM :) Thanks for investigating and fixing this issue!

@FSchumacher
Copy link
Contributor

Looks good to me, but one question to those using influx. Is the change backwards compatible, or has everyone to change their setup afterwards?

@ham1
Copy link
Contributor

ham1 commented Apr 23, 2023

I think the only implication is that a run will now use slightly more storage due to adding a thread tag (but also by not throwing away data points!)

@FSchumacher
Copy link
Contributor

PR has been merged.
Can you test current trunk or next nightly?

@vperaltac
Copy link
Contributor Author

The change should be backwards compatible, no need to modify the setup.
@FSchumacher I just tested the latest trunk and it works great!

@@ -122,8 +122,10 @@ private String createTags(SampleResult sampleResult) {
// remove surrounding quotes and spaces from sample label
String label = StringUtils.strip(sampleResult.getSampleLabel(), "\" ");
String transaction = AbstractInfluxdbMetricsSender.tagToStringValue(label);
String threadName = StringUtils.deleteWhitespace(sampleResult.getThreadName());
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be time-consuming.
Do you really need a thread name?
Would it be workable if we included a "unique thread id"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, if the thread ID is unique at the Test Plan level, it should work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

InfluxDBRawDataListener misses to insert few rows of data
5 participants