www.google.com - Load more results is not triggered

[webcompat-bot]

URL: https://www.google.com/search?source=hp

Browser / Version: Firefox Mobile 63.0
Operating System: Android 8.1.0
Tested Another Browser: Yes

Problem type: Site is not usable
Description: Can't load more results
Steps to Reproduce:
Search in google

From webcompat.com with ❤️

[softvision-oana-arbuzov]

Thanks for the report, I was able to reproduce the issue.
Updated URL: https://www.google.com/search?source=hp&ei=DeM5W5-JFoL-swGEhI_wDg&q=hp&oq=hp&gs_l=mobile-gws-wiz-hp.3...2336.2411..2603...1....153.153.0j1......0....1.......0.8PDh9rwdLBk%3D

Affected area:

<a jsname="oHxHid" href="/search?q=hp&amp;prmd=inmv&amp;ei=beU5W-XXKIijsAG_06LoBw&amp;start=10&amp;sa=N" jsaction="dZgHfd" class="VknLRd" aria-label="More results" role="button" data-hveid="240" data-ved="0ahUKEwilwrLogoDcAhWIESwKHb-pCH0Qqq4CCPABKAI">
	<div class="GNJvt ipz2Oe">
		<span class="kQdGHd z1asCe QFl0Ff">
			<svg focusable="false"
				xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24">
				<path d="M16.59 8.59L12 13.17 7.41 8.59 6 10l6 6 6-6z"></path>
			</svg>
		</span>
		<span class="RVQdVd">More results</span>
	</div>
</a>

Tested with:
Browser / Version: Firefox Nightly 63.0a1 (2018-07-01)
Operating System: Huawei P10 (Android 7.0) - 1080 x 1920 pixels (~432 ppi pixel density), Samsung Galaxy S7 Edge (Android 7.0) - Resolution 1440 x 2560 pixels (~534 ppi pixel density)

Moving to Needsdiagnosis for further investigation.

[karlcow]

@miketaylr @denschub This is probably the UA override consequence.

[denschub]

Yes, can confirm. This is bad.

Clicking the button triggers 4 requests, 3 POST request and 1 GET request. There is a GET request to https://www.google.com/search, which in Chrome returns something that contains more search results and other information, while in Firefox, it just returns an... empty half JSON object.

There are differences in the queries, but... well:

https://www.google.com/search?vet=10ahUKEwiA66XS2ILcAhVOKywKHVCXCwMQxK8CCJgDKAI..i
  &ved=0ahUKEwiA66XS2ILcAhVOKywKHVCXCwMQvvYCCJoDKAQ
  &yv=3
  &q=cat
  &start=10
  &nps=10
  &asearch=arc
  &async=arc_id:srp_4510,ffilt:all,ve_name:MoreResultsContainer,next_id:srp_45,use_ac:true,_id:arc-srp_4510,_pms:qs,_fmt:pc

The vet/ved parameters seem to be significant, and so are the cookies, as I don't get anything back without cookies. The cookies contain more unreadable stuff that looks like some binary encoding. This is going to be fun to debug as well...

[karlcow]

fwiw on RDM with this user agent

Mozilla/5.0 (Linux; Android 4.4.4; en-us; Nexus 5 Build/JOP40D) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2307.2 Mobile Safari/537.36

The more results is working.

[karlcow]

HTTP/2.0 200 OK
version: 201646228
x-content-type-options: nosniff
content-type: text/plain; charset=UTF-8
content-disposition: attachment; filename="f.txt"
trailer: X-Google-GFE-Current-Request-Cost-From-GWS
strict-transport-security: max-age=86400
content-encoding: gzip
date: Tue, 03 Jul 2018 11:00:20 GMT
server: gws
cache-control: private
x-xss-protection: 1; mode=block
x-frame-options: SAMEORIGIN
expires: Tue, 03 Jul 2018 11:00:20 GMT
set-cookie: 1P_JAR=2018-07-03-11; expires=Thu, 02-Aug-2018 11:00:20 GMT; path=/; domain=.google.com
alt-svc: quic=":443"; ma=2592000; v="43,42,41,39,35"
X-Firefox-Spdy: h2

[wisniewskit]

Confirmed, it's our experimental spoof in Fennec nightly. More precisely, the spoof must not be applying to enough Google sub-domains, because when I use a general UA-spoof (not limited to just www.google.com and a few other domains), it works with the same user-agent string that the experiment is using (currently Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) FxQuantum/63.0 AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.0.0 Mobile Safari/537.36). But when the add-on is left to spoof only a few domains, it breaks the results.

I'll try to figure out what other domains need to be spoofed for best results.

Edit: actually, the requests are hitting www.google.com, so the spoofing should be working. However, for some reason the XHR being sent is not sent with the spoofed UA, but the regular Firefox one. So this must be an unexpected logic bug somewhere in the spoofer (or perhaps the technique it's using to spoof requests does not affect XHRs).

Edit #2: it was indeed that the technique the add-on uses (essentially, listening for the Gecko http-on-useragent-request messages) doesn't work. Those messages don't seem to be sent during XHRs, so there is no over-riding going on for them. The message we should be listening to, based on how webRequest.onBeforeSendHeaders works for extensions, is http-on-modify-request, which seems to be working.

Edit #3: alright, I've filed bz1473181 with the fix for this, so I'll close this as a dupe.