Multiple visits with each pageview when "/?pk_campaign" is used

[Glisse1]

Hello guys,

Somewhere after the upgrade to 2.15.0 I got a nasty bug which drove me crazy. When a user is reaching my site in any other way than adwords everything is perfect. But when the visitor comes from adwords (which has /?pk_campaign at the end of url) it tends to generate lots of visits, sometimes as many as pageviews.

The thing that got me going crazy is that i could not reproduce this bug on demo.piwik by directly entering the forum with /?pk_campaign in url. It seems to work fine there.

I have tried everything:

• completeley erased piwik installation from server
• completely erased the database
• clean install 2.15 from scratch and also 2.15.1 beta, even in completely new databases
• lost my data of 2 years in the process, partially recovered monthly stats by manually entering values in database
• I also reverted to the clean javascript code provided by piwik settings with no heartbeat and no userid. On piwik forum the javascript code does seem to look different compared to standard one.

No effect. I have done all the above at least four times. I have concluded that there is something wrong with the way piwik sees a users reaching my site with ?pk_campaign (+1 visits) then going to another page normally (+1 visits) then for instance press back and once again is in the url with ?pk_campaign in it and hence (+1 visits). And so on. On the other hand, if you click the visitor profile you can see all these visit, piwik knows it`s the same visitor, but it just creates new visits for them one after another.

In the provided picture of a visitor reaching my site from google adwords you can see that some of the visits are non-clickable "adwbuc" other are clickable "adwbuc" (originating from my own site) with ?pk_campaign & long gclid when hover.

Having just plain text as the adwords campaign source also happened prior to 2.15 but rarely and also, never created the crazy visits rollercoaster.

Picture of 5 visits created for the same visitor in 7 minutes:

[tsteur]

FYI: There is a setting create_new_visit_when_campaign_changes = 0/1 . I checked demo.piwik.org and the setting is not disabled there, so you it doesn't explain why it was not reproduceable there. Until a solution is found here, it might help to disable this setting as a workaround.

I'm not sure what the "www.prepelitu.ro" is? I had a look at the code and whenever one of the 'referer_keyword', 'referer_name', 'referer_type' changes it creates a new visit.

[Glisse1]

*edited, dont want this details indexed by google

[tsteur]

Are there any different domains/subdomains involved during this process? Eg when navigating to another page?

[Glisse1]

nope, no subdomains on my site...often the user simply refreshes the same landing page because they dont like the long url added by google with all those &clid and characters..

Inserting the setting you suggested in config.ini seems to have done the trick, at least from what i can tell in the past hours..

Although my problem is solved by that setting, the core issue remains: user arrives on site with /?pk_campaing in url and when they navigate to another page - on the same site but no longer having pk_campaing in url - piwik thinks it`s a new referrer.... It iss up to you if there are resources or if it is worth of investigating furthermore, or just close the ticket for now.

Anyway, If I can help you anyhow, give you access to my installation/database, w/e, I would gladly do so.

Thank you for the config.ini solution!

[Glisse1]

Update: seems that although things are better after the config.ini setting, the bug is not completely gone.

i can tell you what happened with this visitor:

• reaching my site via google adwords with "mancare prepelite pret" keyword at 1:59:09 PM
• refreshing the landing page
• going on another page on my site (blue question mark icon) but in this new "visit" instead of the adword keyword the refferer is my own site "www.prepelitu.ro"
• pressing back on the phone and returning to the landing page with pk_campaing in it, the adwords keyword "macare prepelita pret" is once again listed

Total of 3 visits. This is on

[tsteur]

I just had a look at the code and once create_new_visit_when_campaign_changes is disabled it should not create a new visitor on any campaign parameter. No matter if it changes or if it is the same etc. I also can't think of any other reason why it would create new visits there by looking at the code. Just to be sure in case you're running Piwik on multiple servers: Is [Tracker]create_new_visit_when_campaign_changes=0 set on all servers?

Access to the Piwik instance would be helpful as well in case there is actually no config parameter missing on a server. Feel free to email to hello at piwik.org and leave a comment here afterwards in case it goes to spam folder

[Glisse1]

Single server, but I had the setting under [General], did not know that is should have been [Tracker] :)

[tsteur]

Sorry, should have mentioned it in the first place :)

I reckon we should still investigate the issue as disabling the setting is only a work around. I might try to reproduce it soon.

[tsteur]

I couldn't reproduce it so access to your installation might be helpful.

That it creates the first visit and third visit is expected, not the second though.

It creates the first visit because of a new campaign. It creates the third new visit because the last visit had no campaign set (was in first visit). The question is why does it create a new visit for the second pageview. If it didn't create a new visit for the second pageview, it would also not create the third new visit

[Glisse1]

Wiped everything, installed fresh 2.15.0, sent email with details.

[tsteur]

Thx for that! Where does the referrer keyword get set? This is actually the problem. As soon as the campaign name or referrer keyword or the referrer type changes, Piwik creates a new visit which is on purpose.

[Glisse1]

The referrer keyword on this campaign is set by google adwords, the search term on google. Sometimes it`s set, sometimes is not.

[tsteur]

I can finally reproduce it. It is related to the heartbeat timer. For some reason the heart beat timer didn't work in my regular browser window even though it was enabled. Now that I opened it incognito it somehow worked and now I am seeing this problem.

[tsteur]

So there are actually 2 bugs I reckon. Sometimes heartbeat is not sent even though it is enabled and when it is sent it creates many new visits if used in combination with campaigns

[Glisse1]

Yayy, glad it`s a bug :D
I was starting to feel like a fool for wasting your time, and feeling that it was a problem only with me :)

[tsteur]

Even if it was not a bug it's still good to investigate such things so no waste anyway ;)

Re the not working heart beat timer: It's basically impossible to debug this one remotely as heartbeat is based on new Date().getTime() and when stopping the debugger the conditions are not realistic anymore. My assumption is this a bug here maybe: https://github.com/piwik/piwik/blob/2.15.1-b6/js/piwik.js#L3820

• Say at 0.01 sec we setup heart beat with an interval of 10seconds
• At 0.05 sec we start the first tracking request
• At 10.01 sec it will go into this method and calculate lastTrackerRequestTime + configHeartBeatDelay <= now.getTime() which results in (0.05 + 10) < 10.01 as 10.05 is not lower than 10.01 it will not send the ping but instead setup a new heart beat timer here https://github.com/piwik/piwik/blob/2.15.1-b6/js/piwik.js#L3127
• configHeartBeatDelay - (now.getTime() - lastTrackerRequestTime); will result in something like 10 - (10.05 - 0.05) which could evaluate to 0 sec meaning it will not fire the heartbeat up immediately but in 10 seconds https://github.com/piwik/piwik/blob/2.15.1-b6/js/piwik.js#L3131 as delay || configHeartBeatDelay delay could be 0. But as getTime() returns time in milliseconds it is very unlikely 0 so my assumption would be wrong.

Update: I think it was actually just not sending ping requests as I had maybe the debugger open and the browser window was not considered "Active" by heartbeat. So only bug 2 remains that it creates new visits when campaign parameter is used

[Glisse1]

Heartbeat is one of the best piwik feature, and from my point of view, end-user with no knowledge of "under the hood", is working like a charm.. as soon as my visitors are moving the mouse, the visit in live widget turns red (i have it customized, to have red-green color, red for active in last 15 sec, green for active in last 5 minutes)..i have never had problems with heartbeat

[tsteur]

Yeah it seems to actually works fine, was just me thinking it should still send tracking requests when being in the browser debugger but it's actually correct behaviour I guess :)

[tsteur]

@diosmosis can you maybe have a look at this?

[Glisse1]

I have removed the heartbeat code from javascript and it made absolutely no difference.. so, if heartbeat it 's the culprit, then it doesn't even need to be fired-up in the end-user javascript.

[tsteur]

Is it maybe still cached for some people? It's possible that I was wrong and that it is not related to heartbeat. BTW: Which browser are you using when testing? In case you are using Chrome maybe have a look if there's an update as it could be related to this: #9246 (comment)

[Glisse1]

I can confirm that this is not related to the heartbeat code being or not being in javascript code. Tested in clean edge/firefox/internet explorer with cleared cache/cookies, and also incognito chrome which is my main prowser.

[tsteur]

OK thx for that. Can you give us access to your Piwik again? Maybe also to your database and ideally webserver (if possible)?

[Glisse1]

hello, replied to your e-mail with login credentials for piwik and also my webserver`s Control Panel which includes everything, database, file managers, etc.

[Glisse1]

Is anyone using the login details i have e-mailed to look into this bug? I ' m not comfortable to stay for days/weeks with those temporary, unsafe and easy passwords for my site and webserver for nothing.. it`s my online store and i had suffered attacks from direct competition in the past.

[tsteur]

I'm having a look now

[tsteur]

Thank you a lot @Glisse1 this time we found the actual issue. I will let you know once there is a patch available. You can remove the access for us again.

[mattab]

fyi: regressed in #8630 + tagged as Major as it can cause data errors when tracking campaigns without setting a campaign keyword

[Glisse1]

Glad that I could help and you sorted this out

[tsteur]

FYI: I issued a fix for this here: https://github.com/piwik/piwik/pull/9363/files

Are you familiar with applying diffs or patches? There would be a diff here: https://patch-diff.githubusercontent.com/raw/piwik/piwik/pull/9363.diff and a patch here: https://patch-diff.githubusercontent.com/raw/piwik/piwik/pull/9363.patch . It's quite a few changes so probably not easy to apply them manually

If you are not familiar with it, I would recommend to wait for the next beta release after we merged this change.

[Glisse1]

No rush man, I`ve been having this bug for more than a month, I can wait for the official 2.15.1 version. Thank you for working on it!

[adaqus]

@tsteur if someone has no possibility to upgrade to 2.16 (e.g. due to plugin incompatibility), is there possibility to workaround this bug? For instance will providing always campaign keyword (using pk_kwd or similar parameter) can solve the issue?

[tsteur]

I think always providing the keyword would result in many other problems. Only solution I could think of would be to add a random parameter to each referrer so far but it would likely not fix the issue alone as there's still a problem with the keyword. Actually, I think it's not really possible to workaround. Which Piwik version is currently in use?

[adaqus]

The version is 2.15.

[tsteur]

So far I cannot really think of any workaround apart from maybe setting config create_new_visit_when_campaign_changes=0 which is not really a workaround.

Re not possible to update. Did we break anything in 2.16.0 that makes the plugin incompatible?

[adaqus]

By 'no possibility to upgrade' I meant that it is not possible to upgrade to 2.16 as soon as possible, because all needed plugins had to be tested with 2.16 to ensure everything works. It requires significant amount of time. Since there is a need to fix this issue as soon as possible, I was searching for any workaroud/quick fix. Turns out the only solution is upgrading to 2.16, which will be carried out in few days.

[tsteur]

Sweet re upgrading. Sorry I wish there was something we could do but each workaround would likely make it worse. Only possibly thing would otherwise applying only the patch from #9363 but I presume this is not a solution