Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

webconnectivity: inaccurate www.irna.ir measurement #1824

Open
bassosimone opened this issue Oct 18, 2021 · 0 comments
Open

webconnectivity: inaccurate www.irna.ir measurement #1824

bassosimone opened this issue Oct 18, 2021 · 0 comments
Assignees
@bassosimone
Labels
bug Something isn't working data quality priority/low

Comments

@bassosimone
Copy link
Contributor

bassosimone commented Oct 18, 2021 •
edited
Loading

This issue branches off of #1707 (comment). In such an issue, we were comparing the old and the new webconnectivity test helpers. The http://www.irna.ir URL stood out in this comparison, because the old test helper said:

{
  "tcp_connect": {
    "178.216.249.78:80": {
      "status": true,
      "failure": null
    },
    "217.25.48.64:80": {
      "status": true,
      "failure": null
    }
  },
  "http_request": {
    "body_length": 35805,
    "failure": null,
    "title": "IRNA English",
    "headers": {
      // [snip]
    },
    "status_code": 200
  },
  "dns": {
    "failure": null,
    "addrs": [
      "217.25.48.64",
      "178.216.249.78"
    ]
  }
}

while the new test helper said:

{
  "tcp_connect": {
    "178.216.249.78:80": {
      "status": true,
      "failure": null
    },
    "217.25.48.64:80": {
      "status": true,
      "failure": null
    }
  },
  "http_request": {
      // snip
    },
    "status_code": 404
  },
  "dns": {
    "failure": null,
    "addrs": [
      "217.25.48.64",
      "178.216.249.78"
    ]
  }
}

So, the basic take away from this measurement seems to be: the new TH returns 404, the old one 200.

However, digging further into this URL, it appears that the webconnectivity's conclusion is wrong. To illustrate why that happens, let me use the websteps test helper to measure the same URL.

// ./oohelper -websteps -target http://www.irna.ir/en/
{
    "dns": [
        {
            "domain": "www.irna.ir",
            "lookup_host": [
                {
                    "answers": [
                        {
                            "answer_type": "A",
                            "ipv4": "217.25.48.64"
                        },
                        {
                            "answer_type": "A",
                            "ipv4": "178.216.249.78"
                        }
                    ],
                    "engine": "doh",
                    "failure": null,
                    "hostname": "www.irna.ir",
                    "query_type": "A",
                    "resolver_address": "https://dns.google/dns-query",
                    "t": 0.591798353,
                    "started": 0.002539607,
                    "oddity": ""
                },
                {
                    "answers": null,
                    "engine": "doh",
                    "failure": null,
                    "hostname": "www.irna.ir",
                    "query_type": "AAAA",
                    "resolver_address": "https://dns.google/dns-query",
                    "t": 0.591798353,
                    "started": 0.002539607,
                    "oddity": ""
                }
            ]
        }
    ],
    "endpoints": [
        {
            "url": "http://www.irna.ir/en/",
            "network": "tcp",
            "address": "217.25.48.64:80",
            "connect": [
                {
                    "address": "217.25.48.64:80",
                    "failure": null,
                    "operation": "connect",
                    "proto": "tcp",
                    "t": 0.682398684,
                    "started": 0.593380438,
                    "oddity": ""
                }
            ],
            "http_round_trip": [
                {
                    "failure": null,
                    "request": {
                        "method": "GET",
                        "url": "http://www.irna.ir/en/",
                        "headers": {
                            "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
                            "accept-language": "en-US;q=0.8,en;q=0.5",
                            "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36"
                        }
                    },
                    "response": {
                        "code": 302,
                        "headers": {
                            "content-length": "207",
                            "content-type": "text/html; charset=iso-8859-1",
                            "date": "Mon, 18 Oct 2021 07:55:13 GMT",
                            "location": "https://www.irna.ir/en/",                           // <-------------------
                            "server": "Apache"
                        },
                        "body": null,
                        "body_is_truncated": false,
                        "x_body_length": 207,
                        "x_body_is_utf8": true
                    },
                    "t": 0,
                    "started": 0,
                    "oddity": ""
                }
            ]
        },
        {
            "url": "http://www.irna.ir/en/",
            "network": "tcp",
            "address": "178.216.249.78:80",
            "connect": [
                {
                    "address": "178.216.249.78:80",
                    "failure": null,
                    "operation": "connect",
                    "proto": "tcp",
                    "t": 0.68614754,
                    "started": 0.593924847,
                    "oddity": ""
                }
            ],
            "http_round_trip": [
                {
                    "failure": null,
                    "request": {
                        "method": "GET",
                        "url": "http://www.irna.ir/en/",
                        "headers": {
                            "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
                            "accept-language": "en-US;q=0.8,en;q=0.5",
                            "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36"
                        }
                    },
                    "response": {
                        "code": 301,
                        "headers": {
                            "connection": "keep-alive",
                            "content-length": "162",
                            "content-type": "text/html",
                            "date": "Mon, 18 Oct 2021 07:55:13 GMT",
                            "location": "https://www.irna.ir/en/",                           // <-------------------
                            "server": "nginx"
                        },
                        "body": null,
                        "body_is_truncated": false,
                        "x_body_length": 162,
                        "x_body_is_utf8": true
                    },
                    "t": 0,
                    "started": 0,
                    "oddity": ""
                }
            ]
        }
    ]
}

So, we see that we have two endpoints (178.216.249.78:80 and 217.25.48.64:80) and both redirect to https://www.irna.ir/en/. Now, let's do another step and check what we can get when we connect to such a website:

// ./oohelper -websteps -target https://www.irna.ir/en/
{
    "dns": [
        {
            "domain": "www.irna.ir",
            "lookup_host": [
                {
                    "answers": [
                        {
                            "answer_type": "A",
                            "ipv4": "217.25.48.64"
                        },
                        {
                            "answer_type": "A",
                            "ipv4": "178.216.249.78"
                        }
                    ],
                    "engine": "doh",
                    "failure": null,
                    "hostname": "www.irna.ir",
                    "query_type": "A",
                    "resolver_address": "https://dns.google/dns-query",
                    "t": 0.385251655,
                    "started": 0.000177063,
                    "oddity": ""
                },
                {
                    "answers": null,
                    "engine": "doh",
                    "failure": null,
                    "hostname": "www.irna.ir",
                    "query_type": "AAAA",
                    "resolver_address": "https://dns.google/dns-query",
                    "t": 0.385251655,
                    "started": 0.000177063,
                    "oddity": ""
                }
            ]
        },
        {
            "domain": "www.irna.ir",
            "lookup_httpssvc": [
                {
                    "answers": null,
                    "engine": "doh",
                    "failure": "dns_no_answer",
                    "hostname": "www.irna.ir",
                    "query_type": "HTTPS",
                    "resolver_address": "https://dns.google/dns-query",
                    "t": 0.503073972,
                    "started": 0.386212206,
                    "oddity": "dns.lookup.other"
                }
            ]
        }
    ],
    "endpoints": [
        {
            "url": "https://www.irna.ir/en/",
            "network": "tcp",
            "address": "217.25.48.64:443",
            "connect": [
                {
                    "address": "217.25.48.64:443",
                    "failure": null,
                    "operation": "connect",
                    "proto": "tcp",
                    "t": 0.604136343,
                    "started": 0.52453632,
                    "oddity": ""
                }
            ],
            "tls_handshake": [
                {
                    "cipher_suite": "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
                    "failure": null,
                    "negotiated_proto": "h2",
                    "tls_version": "TLSv1.2",
                    "peer_certificates": null,
                    "t": 0,
                    "address": "217.25.48.64:443",
                    "server_name": "www.irna.ir",
                    "alpn": [
                        "h2",
                        "http/1.1"
                    ],
                    "no_tls_verify": false,
                    "oddity": "",
                    "proto": "tcp",
                    "started": 0
                }
            ],
            "http_round_trip": [
                {
                    "failure": null,
                    "request": {
                        "method": "GET",
                        "url": "https://www.irna.ir/en/",
                        "headers": {
                            "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
                            "accept-language": "en-US;q=0.8,en;q=0.5",
                            "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36"
                        }
                    },
                    "response": {
                        "code": 302,                             // <---------------------------
                        "headers": {
                            "content-length": "203",
                            "content-type": "text/html; charset=iso-8859-1",
                            "date": "Mon, 18 Oct 2021 07:58:05 GMT",
                            "location": "https://en.irna.ir/",
                            "server": "Apache"
                        },
                        "body": null,
                        "body_is_truncated": false,
                        "x_body_length": 203,
                        "x_body_is_utf8": true
                    },
                    "t": 0,
                    "started": 0,
                    "oddity": ""
                }
            ]
        },
        {
            "url": "https://www.irna.ir/en/",
            "network": "tcp",
            "address": "178.216.249.78:443",
            "connect": [
                {
                    "address": "178.216.249.78:443",
                    "failure": null,
                    "operation": "connect",
                    "proto": "tcp",
                    "t": 0.614428418,
                    "started": 0.527782532,
                    "oddity": ""
                }
            ],
            "tls_handshake": [
                {
                    "cipher_suite": "TLS_AES_128_GCM_SHA256",
                    "failure": null,
                    "negotiated_proto": "h2",
                    "tls_version": "TLSv1.3",
                    "peer_certificates": null,
                    "t": 0,
                    "address": "178.216.249.78:443",
                    "server_name": "www.irna.ir",
                    "alpn": [
                        "h2",
                        "http/1.1"
                    ],
                    "no_tls_verify": false,
                    "oddity": "",
                    "proto": "tcp",
                    "started": 0
                }
            ],
            "http_round_trip": [
                {
                    "failure": null,
                    "request": {
                        "method": "GET",
                        "url": "https://www.irna.ir/en/",
                        "headers": {
                            "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
                            "accept-language": "en-US;q=0.8,en;q=0.5",
                            "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36"
                        }
                    },
                    "response": {
                        "code": 404,                             // <---------------------------
                        "headers": {
                            "content-language": "en",
                            "content-length": "1922",
                            "content-type": "text/html;charset=UTF-8",
                            "date": "Mon, 18 Oct 2021 07:58:06 GMT",
                            "server": "nginx"
                        },
                        "body": null,
                        "body_is_truncated": false,
                        "x_body_length": 1922,
                        "x_body_is_utf8": true
                    },
                    "t": 0,
                    "started": 0,
                    "oddity": "http.status.404"
                }
            ]
        }
    ]
}

This second step is enough to clarify what is happening. The 217.25.48.64:443 endpoint returns 302 and redirects to https://en.irna.ir/. The 178.216.249.78:443 endpoint returns 404.

Bottom line: testing each endpoint helps to spot these weird cases and helps to figure out misconfiguration. In turn, by doing that, we increase the data quality because we're less likely to say "anomaly". If the probe and the TH both conclude that 178.216.249.78:443 returns 404, there is, in fact, no anomaly.
@bassosimone bassosimone self-assigned this Oct 18, 2021
@bassosimone bassosimone added the bug Something isn't working label Oct 18, 2021
@bassosimone bassosimone changed the title webconnectivity: incomplete www.irna.ir measurement webconnectivity: inaccurate www.irna.ir measurement Oct 18, 2021
This was referenced Oct 18, 2021
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bassosimone bassosimone

Labels
bug Something isn't working data quality priority/low
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@bassosimone