Consistent 403 at url that works in browser

[stevenvachon]

request(
    "http://validator.w3.org/check?uri=https%3A%2F%2Fwww.svachon.com%3A443%2F",
    function(error, response) {
        console.log(response.statusCode);  //-> 403
    }
);

I'm not much of a server guy, so I'm not quite sure how to debug this. I'm not using a proxy, but I am running a firewall (?).

[papandreou]

Reproducible. The response body of the 403 response says (snip):

        <p> Please identify your software properly by providing the
          HTTP User-Agent string.  Default from an underlying library
          (eg Java, Ruby, Coda, libwww) is not sufficiently unique in
          identifying your software.  A good practice is to include a
          URI to more information about your service or software.
          </p>

And indeed, specifying any User-Agent header does allow you through:

require('request')({
    url: "http://validator.w3.org/check?uri=https%3A%2F%2Fwww.svachon.com%3A443%2F",
    headers: {'User-Agent': 'foo'}
}, function(error, response) {
    console.log(response.statusCode);  //-> 200
})

But you should of course respect their terms and provide a value that allows them to identify your software.

[decal]

You can also set the uri GET query variable to 'referer' and assign the URL-decoded URI to the HTTP request header Referer: .. This might mitigate encoding problems in the future since you can pass a literal URI as the Referer: header value, just as you would type it in a browser address bar.

[stevenvachon]

Thanks @papandreou, that fixed it!

@decal But don't many requests not contain referrers? What kind of encoding problems? Also, could you perhaps provide an example?