scalaj-http doesn't set default "User-Agent" value into HTTP request headers.

[kypeli]

scala> val html = Http("http://www.questionablecontent.net").asString
scalaj.http.HttpException: 403: Forbidden
at scalaj.http.Http$Request.process(Http.scala:125)
at scalaj.http.Http$Request.apply(Http.scala:102)
at scalaj.http.Http$Request.asString(Http.scala:151)
at .(:11)
at .()
at .(:7)
at .()
at $print()
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
...

kypeli@iceberg:~$ curl -I http://www.questionablecontent.net
HTTP/1.1 200 OK
Server: cloudflare-nginx
Date: Sun, 21 Jul 2013 11:40:50 GMT
Content-Type: text/html
Connection: keep-alive
X-Powered-By: PHP/5.4.13
Set-Cookie: __cfduid=dd4e453443f2d6171976787db3fdf55c81374406850; expires=Mon, 23-Dec-2019 23:50:00 GMT; path=/; domain=.questionablecontent.net
CF-RAY: 9179e60760306dc

[fntz]

Set headers for request

 val html = Http("http://www.questionablecontent.net").header("User-Agent", "Opera").asString

[kypeli]

I see, thanks! That indeed works. So it seems the issue here was in fact that the server didn't accept requests without the User-Agent set.

In that case I would have to ask if it's expected behavior that scalaj-http doesn't set the User-Agent by default? And if it is, should it? I think it should.

[hoffrocket]

the User-Agent is usually set to the version of java you're using. (scalaj-http just wraps the built in java http library). Some web servers will block access based on User-Agent to prevent scraping. That's probably what's going on here.

scala> Http("http://httpbin.org/headers").asString
res1: String = 
{
  "headers": {
    "User-Agent": "Java/1.6.0_33",
    "Host": "httpbin.org",
    "Connection": "close",
    "Accept": "text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2"
  }
}