[2.4.x] Upgrade async-http-client to version 1.9.29 by marcospereira · Pull Request #4826 · playframework/playframework

marcospereira · 2015-07-09T04:18:32Z

This PR backports #4778 to branch 2.4.x.

brianwawok · 2015-07-09T15:22:42Z

My oauth signatures totally broke in a 2.3 to 2.4 migration. Tried this fix by manually updating my async client, and it did not help me at all, still getting invalid oauth returned from multiple oauth 1 providers.

wsargent · 2015-07-09T15:34:46Z

@brianwawok can you put together a test case reproducing the problem?

brianwawok · 2015-07-09T16:50:18Z

@wsargent not sure I can provide a test case scala-test style, as I am not super familiar with what is going on in the encoding level of http async. Here is a test case though hitting up the etsy api.. (https://www.etsy.com/developers/)

I cannot use the inbuilt play oauth thingy because I need to pass params (perms here), so I can't do it the way twitter does it in the docs.

    val redirectUrl = "http://localhost/processToken"
    val etsyKey = ConsumerKey("myapiKey", "mysharedSecret") //replace with etsy api keys
    val requestToken = RequestToken("", "") //this is really blank for a request token request
    val perms = "listings_r listings_w listings_d transactions_r transactions_w shops_rw"   
    val requestUrl = "https://openapi.etsy.com/v2/oauth/request_token"
    val nonce = "abcdefg" //really do something better

    WS.url(requestUrl).sign(OAuthCalculator(etsyKey, requestToken))
      .withRequestTimeout(10000)
      .withHeaders("Content-Type" -> "application/x-www-form-urlencoded", "Accept" -> "application/json")
      .withQueryString("scope" -> perms)
      .post(
        s"oauth_callback=$redirectUrl&" +
          s"oauth_nonce=${nonce}&" +
          s"oauth_timestamp=${DateTime.now(DateTimeZone.UTC).getMillis / 1000}&" +
          s"oauth_version=1.0")

Works in 2.3, in 2.4 I get back from etsy api

 oauth_problem=signature_invalid

marcospereira · 2015-07-09T17:04:14Z

@brianwawok this looks good enough to me. I will try to use your scenario to replicate the problem and see if it is a problem with Play or async-http-client. Can't do it right now, but will try later today.

Thanks!

coolsuntraveler · 2015-07-09T17:56:28Z

I also experience the same problem. My oauth 1.0a request to Yelp APIs used to work in 2.3, and now I get invalid signature error. My code is as follows:

WS.url("http://api.yelp.com/v2/search?radius_filter=20000&term=BJ%27s+Restaurant+%26+Brewhouse&location=401+The+Bridge+Street.+Huntsville%2C+AL.+35086")
     .sign(OAuthCalculator(ConsumerKey(YELP_CONSUMER_KEY, YELP_CONSUMER_SECRET), RequestToken(YELP_TOKEN, YELP_TOKEN_SECRET)))
     .get

I asked this question on StackOverflow. http://stackoverflow.com/questions/31253491/oauth-1-0a-stops-working-after-upgrading-from-play-2-3-to-play-2-4. Thanks.

marcospereira · 2015-07-10T05:55:31Z

@brianwawok and @coolsuntraveler,

Directly using async-http-client also results in a 401 Unauthorized, with oauth_problem=signature_invalid. Here is the code:

import org.joda.time._

import com.ning.http.client._
import com.ning.http.client.oauth._

val redirectUrl = "http://localhost/processToken"
val consumerKey = new ConsumerKey("...", "...") //replace with etsy api keys
val requestToken = new RequestToken("", "") //this is really blank for a request token request
val perms = "listings_r listings_w listings_d transactions_r transactions_w shops_rw"   
val requestUrl = "https://openapi.etsy.com/v2/oauth/request_token"
val nonce = "abcdefg" //really do something better

val client = new AsyncHttpClient()

val response = client
  .preparePost(requestUrl)
  .setSignatureCalculator(new OAuthSignatureCalculator(consumerKey, requestToken))
  .setRequestTimeout(10000)
  .addHeader("Content-Type", "application/x-www-form-urlencoded")
  .addHeader("Accept", "application/json")
  .addQueryParam("scope", perms)
  .addFormParam("oauth_callback", requestUrl)
  .addFormParam("oauth_nonce", nonce)
  .addFormParam("oauth_timestamp", String.valueOf(DateTime.now(DateTimeZone.UTC).getMillis / 1000))
  .addFormParam("oauth_version", "1.0")
  .execute()

Which generates a request with the following headers:

POST /v2/oauth/request_token?scope=listings_r%20listings_w%20listings_d%20transactions_r%20transactions_w%20shops_rw HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Accept: application/json
Authorization: OAuth oauth_consumer_key="...", oauth_token="", oauth_signature_method="HMAC-SHA1", oauth_signature="..., oauth_timestamp="1436506301", oauth_nonce="...", oauth_version="1.0"
Content-Length: 139
Connection: keep-alive
Host: openapi.etsy.com
User-Agent: AHC/1.0

And the following response:

HTTP/1.1 401 Unauthorized
Server: Apache
X-Etsy-Request-Uuid: 32zDPZB45mWlLHkD1pAh8BiiCYTT
X-OAuth-Problem-Definitions: http://etsy.me/oauth-problem-reporting
WWW-Authenticate: OAuth realm="Etsy"
X-Error-Detail: oauth_problem=signature_invalid&amp;debug_sbs=POST&amp;https%3A%2F%2Fopenapi.etsy.com%2Fv2%2Foauth%2Frequest_token&amp;oauth_callback%3Dhttps%253A%252F%252Fopenapi.etsy.com%252Fv2%252Foauth%252Frequest_token%26oauth_consumer_key%3D...%26oauth_nonce%3Dabcdefg%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1436506301%26oauth_token%3D%26oauth_version%3D1.0%26scope%3Dlistings_r%2520listings_w%2520listings_d%2520transactions_r%2520transactions_w%2520shops_rw
Cache-Control: private
Content-Length: 488
X-Cnection: close
Content-Type: text/plain;charset=UTF-8
Date: Fri, 10 Jul 2015 05:31:41 GMT
Connection: close
Set-Cookie: uaid=uaid%3DQgqDxT7lz51ImUK5u8HHXcGk2bud%26_now%3D1436506301%26_slt%3DdY9t1Q5d%26_kid%3D1%26_ver%3D1%26_mac%3DevFN7j9_u7ZHd2bBDStKgzPVSBi_BdVyReUl7P6CReo.; expires=Mon, 08-Aug-2016 21:50:01 GMT; Max-Age=34186700; path=/; domain=.etsy.com; httponly

(Of course, I've omitted sensible information here).

So, we either have a problem with async-http-client itself, or the request has some problem. I don't know oAuth 1.0 enough. Do you think there is some problem in the code above? The following request returns a 200 OK:

val response = client
  .preparePost(requestUrl)
  .setSignatureCalculator(new OAuthSignatureCalculator(consumerKey, requestToken))
  .setRequestTimeout(10000)
  .addHeader("Content-Type", "application/x-www-form-urlencoded")
  .addHeader("Accept", "application/json")
  .addQueryParam("scope", perms)
  .execute()

What do you guys think?

brianwawok · 2015-07-10T13:33:45Z

@marcospereira I do not know if the code above is valid oauth (like you oauth1 makes me scratch my head a little).. But I do know that it worked in Play 2.3, then stopped in 2.4 (which it looks like to blame is the async-http-client change). I would be curious if the code above works with an old async-http-client, but I suspect it would.

Your request that got a 200 response looks the exact same, but with no form params? Which means it wouldn't really work, as we have to at least set the oauth_callback up for our next request, or the client would never make it back to our server after approving us.

So it seems like there has been some regression in how async-http-client signs form params...

brianwawok · 2015-07-10T13:51:29Z

This is what play 2.3 does

Request DefaultHttpRequest(chunked: false)
POST /v2/oauth/request_token?scope=listings_r%20listings_w%20listings_d%20transactions_r%20transactions_w%20shops_rw HTT
P/1.1
Host: openapi.etsy.com
Content-Type: application/x-www-form-urlencoded
Accept: application/json
Authorization: OAuth oauth_callback="http%3A%2F%2Flocalhost%3A9000%2Fportal%2Ftoken%2Fprocess%2FEtsy", oauth_consumer_ke
y="xxxxxx", oauth_nonce="XLo~K6%21AjSZ8zgzU, oauth_timestamp%3D1436535827", oauth_signature="Jcn9AvdwoDX
tL7IyQ22XgPAsn74%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1436535827", oauth_version="1.0"
Connection: keep-alive
User-Agent: NING/1.0
Content-Length: 135

Response DefaultHttpResponse(chunked: false)
HTTP/1.1 200 OK
Server: Apache
Set-Cookie: uaid=uaid%3DXXXXXXXXXX%26_now%3D14XXXXXXX824%26_slt%3Dxbx-rXX1%26_kid%3D1%26_ver%3D1%26_mac%
XXXXXXXXXXXXXXXXXXXXXX_zwsCu8jeGB-G5VY.; expires=Tue, 09-Aug-2016 06:02:04 GMT; Max-Age=34186700; path=/; domain=
.etsy.com; httponly
X-Etsy-Request-Uuid: hE4-IHF_1cMo3JXXXXXXXXXKO
Cache-Control: private
X-Cnection: close
Content-Type: application/x-www-form-urlencoded
Content-Length: 394
Accept-Ranges: bytes
Date: Fri, 10 Jul 2015 13:43:44 GMT
Via: 1.1 varnish
Connection: keep-alive
X-Served-By: cache-ord1720-ORD
X-Cache: MISS
X-Cache-Hits: 0
X-Timer: S143XXXXXXXX4.285060,VS0,VE84

coolsuntraveler · 2015-07-10T16:09:43Z

@brianwawok, can you please tell me how to get a http request printout like your last comment?

brianwawok · 2015-07-10T16:45:52Z

@coolsuntraveler put logging to debug, com.ning.http.client will do it for you

coolsuntraveler · 2015-07-10T18:57:10Z

Thanks @brianwawok. I compared yelp api requests for Play 2.3 and 2.4, and they look the same to me, other than the values of nonce, timestamp, and signature.

I tried to use async-http-client directly, and it doesn't work for even 1.5.0 and 1.8.0. Either I am using it wrong, or oauth never worked for async-http-client. This is the code for 1.8.0.

val consumerKey = new ConsumerKey(YELP_CONSUMER_KEY, YELP_CONSUMER_SECRET) 
val requestToken = new RequestToken(YELP_TOKEN, YELP_TOKEN_SECRET)
val requestUrl = "https://api.yelp.com/v2/search"

val client = new AsyncHttpClient()
val fr = client
    .prepareGet(requestUrl)
    .setSignatureCalculator(new OAuthSignatureCalculator(consumerKey, requestToken))
    .addQueryParameter("radius_filter", "20000")
    .addQueryParameter("term", "BJ%27s%20Restaurant%20%26%20Brewhouse")
    .addQueryParameter("location", "2624%20S.%20Shackleford%20Rd.%20Little%20Rock%2C%20AR.%2072205")
    .execute

coolsuntraveler · 2015-07-11T01:23:28Z

I found if I don't encode the parameters, Yelp API works with async-http-client 1.9.29.

val consumerKey = new ConsumerKey(YELP_CONSUMER_KEY, YELP_CONSUMER_SECRET) 
val requestToken = new RequestToken(YELP_TOKEN, YELP_TOKEN_SECRET) 
val requestUrl = "https://api.yelp.com/v2/search"

val client = new AsyncHttpClient()
val fr = client
    .prepareGet(requestUrl)
    .setSignatureCalculator(new OAuthSignatureCalculator(consumerKey, requestToken))
    .addQueryParam("radius_filter", "20000")
    .addQueryParam("term", "BJ's Restaurant & Brewhouse")
    .addQueryParam("location", "2624 S. Shackleford Rd. Little Rock, AR. 72205")
    .execute

But this still gets invalid signature in Play 2.4.

WS.url("http://api.yelp.com/v2/search")
  .withQueryString(("radius_filter", "20000"),
                   ("term", "BJ's Restaurant & Brewhouse"),
                   ("location", "2624 S. Shackleford Rd. Little Rock, AR. 72205"))
  .sign(OAuthCalculator(ConsumerKey(YELP_CONSUMER_KEY,YELP_CONSUMER_SECRET),
                        RequestToken(YELP_TOKEN,YELP_TOKEN_SECRET))).get

brianwawok · 2015-07-13T18:03:32Z

@marcospereira

so to further make things interesting, I did this in play 2.4.

Went to http://oauth.googlecode.com/svn/code/javascript/example/signature.html, put in all my stuff.
Ran some code that looked like this:

    val requestUrl = "https://openapi.etsy.com/v2/oauth/request_token"
    val queryString="""oauth_callback=http%3A%2F%2Flocalhost%3A9000%2Fxxxxxxxx&oauth_consumer_key=xxxxxxx&oauth_nonce=xxxxxxx&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1436810252&oauth_version=1.0&scope=listings_r%20listings_w%20listings_d%20transactions_r%20transactions_w%20shops_rw"""

    WS.url(requestUrl + "?" + queryString)
      .withRequestTimeout(10000)
      .withHeaders("Content-Type" -> "application/x-www-form-urlencoded", "Accept" -> "application/json")
      .withHeaders("Authorization" -> """OAuth realm="",oauth_callback="http%3A%2F%2Flocalhost%3A9000%2Fxxxxxxxxxx",oauth_version="1.0",oauth_consumer_key="xxxxxxxx",oauth_timestamp="1436810252",oauth_nonce="9FTZEbfg7I7",oauth_signature_method="HMAC-SHA1",oauth_signature="xxxxxxx"""")
      .post("")

And it worked.

So something is wrong with the signature calculator... hand calculator signature "works"

marcospereira · 2015-07-15T07:05:47Z

Ok, after some investigation, I don't think this is a regression in either async-http-client or play itself, but just the lack of backward compatibility. Turns out async-http-client handles oauth_nonce, oauth_timestamp and oauth_version by itself and you don't need to specify them anymore:

https://github.com/AsyncHttpClient/async-http-client/blob/async-http-client-1.9.29/src/main/java/com/ning/http/client/oauth/OAuthSignatureCalculator.java#L85-L86

This is the code that works as expected:

    val redirectUrl = "http://localhost/processToken"
    val etsyKey = ConsumerKey("...", "...") //replace with etsy api keys
    val requestToken = RequestToken("", "") //this is really blank for a request token request
    val perms = "listings_r listings_w listings_d transactions_r transactions_w shops_rw"
    val requestUrl = "https://openapi.etsy.com/v2/oauth/request_token"

    val client = NingWSClient()

    client.url(requestUrl)
      .sign(OAuthCalculator(etsyKey, requestToken))
      .withRequestTimeout(10000)
      .withHeaders("Content-Type" -> "application/x-www-form-urlencoded", "Accept" -> "application/json")
      .withQueryString("scope" -> perms)
      .post(s"oauth_callback=$redirectUrl")

It also generates signatures which are compatible with the page linked by @brianwawok. Manually posting oauth_nonce, oauth_timestamp and oauth_version was causing a invalid signature base string, since these parameters were duplicated.

@coolsuntraveler I was not able to reproduce your scenario. Could you please submit the logs generated when you execute the code samples you posted here?

brianwawok · 2015-07-15T14:39:45Z

@marcospereira

I agree this is not a http client thing, as I played with older clients with no success.

Also agree we don't need to add oauth stuff, as it gets auto added now. (I think maybe it got auto added before, but there was logic to not auto add it if it was already in the request).

However the big problem - I cannot get your example to work in Play WS syntax. Why does this not work?

 val redirectUrl = "http://localhost/processToken"
    val etsyKey = ConsumerKey("...", "...") //replace with etsy api keys
    val requestToken = RequestToken("", "") //this is really blank for a request token request
    val perms = "listings_r listings_w listings_d transactions_r transactions_w shops_rw"
    val requestUrl = "https://openapi.etsy.com/v2/oauth/request_token"

    WS.url(requestUrl)
      .sign(OAuthCalculator(etsyKey, requestToken))
      .withRequestTimeout(10000)
      .withHeaders("Content-Type" -> "application/x-www-form-urlencoded", "Accept" -> "application/json")
      .withQueryString("scope" -> perms)
      .post(s"oauth_callback=$redirectUrl")

marcospereira · 2015-07-15T15:28:31Z

@brianwawok what do you mean by not working?

Adapting the code sample I've sent to use WS instead of a WSClient worked as expected to me:

    val redirectUrl = "http://localhost/processToken"
    val etsyKey = ConsumerKey("...", "...") //replace with etsy api keys
    val requestToken = RequestToken("", "") //this is really blank for a request token request
    val perms = "listings_r listings_w listings_d transactions_r transactions_w shops_rw"
    val requestUrl = "https://openapi.etsy.com/v2/oauth/request_token"

    WS.url(requestUrl)
      .sign(OAuthCalculator(etsyKey, requestToken))
      .withRequestTimeout(10000)
      .withHeaders("Content-Type" -> "application/x-www-form-urlencoded", "Accept" -> "application/json")
      .withQueryString("scope" -> perms)
      .post(s"oauth_callback=$redirectUrl")

This generates a 200 OK. I've notice that you called the sign method twice and also, keep in mind I'm using async-http-client 1.9.29. Make sure you are using this version too.

brianwawok · 2015-07-15T16:34:41Z

@marcospereira Got it, with removing oauth stuff AND upgrading async to 1.9.29, life is good.

Maybe worth a mention in the migration docs that sending your own oauth params will not result in an invalid signaute? Seems kind of weird still, to just calc the signature wrong on including some oauth params.

coolsuntraveler · 2015-07-15T17:48:27Z

@marcospereira
This is the log from Play 2.4.2. Please let me know if there is anything else you need. Thanks.

Request DefaultHttpRequest(chunked: false)
GET /v2/search?radius_filter=20000&term=BJ%27s%20Restaurant%20%26%20Brewhouse&location=2624%20S.%20Shackleford%20Rd.%20Little%20Rock%2C%20AR.%2072205 HTTP/1.1
Authorization: OAuth oauth_consumer_key="censored", oauth_token="censored", oauth_signature_method="HMAC-SHA1", oauth_signature="P%2FBpRPMb94AXzRjiIh%2B38T%2FEDMI%3D", oauth_timestamp="1436981935", oauth_nonce="WEDvc6JQYzAm4tUorkIY7A%3D%3D", oauth_version="1.0"
Connection: keep-alive
Host: api.yelp.com
Accept: */*
User-Agent: AHC/1.0

Response DefaultHttpResponse(chunked: false)
HTTP/1.1 400 Bad Request
Date: Wed, 15 Jul 2015 17:38:54 GMT
Server: Apache
X-Node: web33-uswest1aprod, api_com
Cache-Control: private
Set-Cookie: bse=6d6952c7fdc5bd32dc3d1c3f5ee4ade5; Domain=.yelp.com; Path=/; HttpOnly
Content-Length: 599
Vary: User-Agent
Content-Type: application/json; charset=UTF-8
X-Mode: ro
X-Proxied: extlb1-uswest1aprod

marcospereira · 2015-07-15T17:52:32Z

@coolsuntraveler could you please also post the response body?

marcospereira · 2015-07-15T17:53:52Z

@coolsuntraveler also, did you tried with async-http-client 1.9.29?

coolsuntraveler · 2015-07-15T18:05:21Z

Response body:

{"error": {"text": "Signature was invalid", "id": "INVALID_SIGNATURE", "description": "Invalid signature. Expected signature base string: GET\u0026http%3A%2F%2Fapi.yelp.com%2Fv2%2Fsearch\u0026location%3D2624%2520S.%2520Shackleford%2520Rd.%2520Little%2520Rock%252C%2520AR.%252072205%26oauth_consumer_key%3Dcensored%26oauth_nonce%3DWEDvc6JQYzAm4tUorkIY7A%253D%253D%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1436981935%26oauth_token%3Dicensored%26oauth_version%3D1.0%26radius_filter%3D20000%26term%3DBJ%2527s%2520Restaurant%2520%2526%2520Brewhouse"}}

I am not familiar with this. I tried with Play 2.4.2. How do I upgrade async-http-client to 1.9.29? Thanks.

marcospereira · 2015-07-15T18:06:41Z

@coolsuntraveler add this line to your build.sbt file:

libraryDependencies += "com.ning" % "async-http-client" % "1.9.29"

coolsuntraveler · 2015-07-15T18:22:00Z

@marcospereira It works after upgrading to 1.9.29. Thanks for your help!

richdougherty · 2015-07-16T17:31:58Z

LGTM. @wsargent, do you want to review?

marcospereira · 2015-07-28T03:41:29Z

@richdougherty and @wsargent any news here?

dotta · 2015-08-20T17:19:44Z

framework/src/play-java-ws/src/main/java/play/libs/ws/ning/NingWSCookie.java

This seems fishy since it's casing a long into an int

Mmm, I realise that since none pointed this out I must be missing something :-)

I don't like it either, but this was done to keep backward compatibility.

The biggest thing I'd be worried about here is whether this has changed from seconds to milliseconds. You can't use an int for a cookie max age in milliseconds because the longest max age you would be able to specify then would be 24 days. But if you change it to long, you can use milliseconds.

Based on this test and also on the fact that max-age header is represented in seconds, I think (although I'm not sure) AHC is expecting seconds here.

Maybe this is an issue that doesn't exist, but what if the returned Long is greater than Int.MaxValue?

One more question about binary compatibility, is there a compelling reason for merging this in 2.4.x?

Maybe this is an issue that doesn't exist, but what if the returned Long is greater than Int.MaxValue?

Said otherwise, I'm wondering why not having Long.min(ahcCookie.getMaxAge(), Integer.MAX_VALUE).intValue() to avoid any potential overflow.

One more question about binary compatibility, is there a compelling reason for merging this in 2.4.x?

It fixes some problems with oAuth1 signatures.

Said otherwise, I'm wondering why not having Long.min(ahcCookie.getMaxAge(), Integer.MAX_VALUE).intValue() to avoid any potential overflow.

We can do that, which is choose between overflowing or returning the wrong value. I prefer to keep this as a int because it is changing less things.

We can do that, which is choose between overflowing or returning the wrong value. I prefer to keep this as a int because it is changing less things.

Ok. My rationale was that if ahcCookie.getMaxAge() would return (Integer.MAX_VALUE + 1), I thought it was better to return Integer.MAX_VALUE, rather than Integer.MIN_VALUE because it overflowed.

I got that. My point was that returning the "wrong/unexpected value" (Integer.MAX_VALUE) is no better than the overflow. Anyway, I don't have a strong opinion here. Both have their own problems.

marcospereira · 2015-08-20T21:00:16Z

@dotta, thanks for reviewing this PR. :-)

brianwawok · 2015-08-21T12:54:01Z

@dotta oauth is broke without this merge, isn't a regression a compelling reason to merge this into 2.4?

jroper · 2015-08-23T23:39:54Z

@brianwawok introducing another regression (breaking binary compatibility) to solve a regression is not a good thing.

dotta · 2015-10-07T08:06:10Z

This PR has stayed open for several months now, and there hasn't been any further follow up since August 21th. If someone is going to investigate how oauth can be fixed without breaking binary compatibility, please feel free to reopen the PR or create a new one.

manuelbernhardt · 2015-12-22T09:25:27Z

Fun fact: I will need to use the workaround outlined in this PR / issue and reference this bug in the upcoming release of Reactive Web Applications. It's not pretty but it doesn't look like I have an alternative...

dotta · 2015-12-22T14:12:42Z

@manuelbernhardt Maybe Play 2.5 will be released before your book is published?

manuelbernhardt · 2015-12-22T14:39:51Z

@dotta It will be a tie, the book will remain on 2.4. It will likely come out in March and will go to production (= all the processing needed to get it to print) very soon, which means the content cannot be changed anymore. Also there are a few dependencies on third-party libraries, so waiting for 2.5 would mean waiting for at least 3-4 months longer, which I'm not too much interested in.

marcospereira mentioned this pull request Jul 9, 2015

WS adds percent encoding when signing with OAuth #4778

Merged

Upgrade async-http-client to version 1.9.29

f844188

dotta reviewed Aug 20, 2015
View reviewed changes

benmccann changed the title ~~Upgrade async-http-client to version 1.9.29~~ [2.4.x] Upgrade async-http-client to version 1.9.29 Sep 25, 2015

dotta closed this Oct 7, 2015

marcospereira deleted the upgrade-ahc-play-2.4.x branch February 19, 2016 02:57

Uh oh!

Conversation

marcospereira commented Jul 9, 2015

Uh oh!

brianwawok commented Jul 9, 2015

Uh oh!

wsargent commented Jul 9, 2015

Uh oh!

brianwawok commented Jul 9, 2015

Uh oh!

marcospereira commented Jul 9, 2015

Uh oh!

coolsuntraveler commented Jul 9, 2015

Uh oh!

marcospereira commented Jul 10, 2015

Uh oh!

brianwawok commented Jul 10, 2015

Uh oh!

brianwawok commented Jul 10, 2015

Uh oh!

coolsuntraveler commented Jul 10, 2015

Uh oh!

brianwawok commented Jul 10, 2015

Uh oh!

coolsuntraveler commented Jul 10, 2015

Uh oh!

coolsuntraveler commented Jul 11, 2015

Uh oh!

brianwawok commented Jul 13, 2015

Uh oh!

marcospereira commented Jul 15, 2015

Uh oh!

brianwawok commented Jul 15, 2015

Uh oh!

marcospereira commented Jul 15, 2015

Uh oh!

brianwawok commented Jul 15, 2015

Uh oh!

coolsuntraveler commented Jul 15, 2015

Uh oh!

marcospereira commented Jul 15, 2015

Uh oh!

marcospereira commented Jul 15, 2015

Uh oh!

coolsuntraveler commented Jul 15, 2015

Uh oh!

marcospereira commented Jul 15, 2015

Uh oh!

coolsuntraveler commented Jul 15, 2015

Uh oh!

richdougherty commented Jul 16, 2015

Uh oh!

marcospereira commented Jul 28, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marcospereira commented Aug 20, 2015

Uh oh!

brianwawok commented Aug 21, 2015

Uh oh!

jroper commented Aug 23, 2015

Uh oh!