Same website on IPv6 and IPv4 not detecting difference on IPv4 HTTP reachablility if IPv4 HTTPS is reachable

[bwbroersma]

https://internet.nl/site/www.gelderland.nl/3200502/#control-panel-8 (also gelderland.nl)

Credits for @janwillemstegink for finding it (also see https://securityheaders.hostingtool.org/index.php?url=gelderland.nl).

Confirmed with this bash code on HTTP:

UA="Mozilla/5.0 (compatible; internetnl/1.9.0; +https://internet.nl/about/)"
FQDN_URL="http://www.gelderland.nl"
diff \
    <(curl -4 -m10 --http1.1 -sfA "$UA" "$FQDN_URL" -o/dev/null -D-) \
    <(curl -6 -m10 --http1.1 -sfA "$UA" "$FQDN_URL" -o/dev/null -D-)

0a1,8
> HTTP/1.1 301 Moved Permanently
> Server: awselb/2.0
> Date: Thu, 20 Mar 2025 16:39:38 GMT
> Content-Type: text/html
> Content-Length: 134
> Connection: keep-alive
> Location: https://www.gelderland.nl:443/
> 

Note the domain has two A and AAAA records:

$ dig +noall +answer gelderland.nl A

gelderland.nl.          184     IN      A       35.71.165.93
gelderland.nl.          184     IN      A       52.223.51.112

Both are not responding on IPv4:

FQDN="gelderland.nl"
curl -6 --resolve "$FQDN:80:35.71.165.93" -m3 -sSf "http://$FQDN" -o/dev/null -D-

curl: (28) Operation timed out after 3001 milliseconds with 0 bytes received

FQDN="gelderland.nl"
curl -6 --resolve "$FQDN:80:52.223.51.112" -m3 -sSf "http://$FQDN" -o/dev/null -D-

curl: (28) Operation timed out after 3001 milliseconds with 0 bytes received

Code issue is here:

Internet.nl/checks/tasks/ipv6.py

Lines 495 to 535 in 77dde1c

	def simhash(url, task=None):
	"""
	Connect on both IPv4 and IPv6 and see if the same content is served.
	First try to connect over HTTP. If that fails for one of the addresses
	try HTTPS for both.
	It uses SequenceMatcher to compare the contents.
	"""
	def strip_irrelevant_html(html):
	"""
	Strip irrelevant HTML for correct comparison.
	This currently strips nonces from script and style tags.
	"""
	soup = BeautifulSoup(html, "html.parser")
	for tag in soup.select(",".join([f"{t}[nonce]" for t in ("script", "style")])):
	del tag["nonce"]
	hidden_tags = soup.find_all("input", {"name": "__VIEWSTATE"})
	for tag in hidden_tags:
	tag.extract()
	try:
	return soup.prettify("latin-1")
	except RecursionError:
	return html
	simhash_score = scoring.WEB_IPV6_WS_SIMHASH_FAIL
	distance = SIMHASH_NOT_CALCULABLE
	v4_response = None
	v6_response = None
	for port in [80, 443]:
	try:
	v4_response = http_get_af(hostname=url, port=port, af=socket.AF_INET, https=port == 443)
	v6_response = http_get_af(hostname=url, port=port, af=socket.AF_INET6, https=port == 443)
	break
	except requests.RequestException:
	pass

If HTTP on IPv4 is an error, HTTP on IPv6 is never checked, and the HTTPS compare is done, while no HTTP on IPv4 while HTTP on IPv6 should be detected as a negative result.

[bwbroersma]

Seems fixed:
gelderland.nl now correctly has a ❌ : https://dev1.internet.nl/site/gelderland.nl/31798/

@janwillemstegink: IPv6 only websites are still correctly ✅ 100%:
https://dev1.internet.nl/site/ipv6.internet.nl/31796/
https://dev1.internet.nl/site/ipv6limited.workingornot.org/31797/