Investigate consistency issue between /latest and /next/10

[moulliet]

In a replicated channel, @chriskessel has found the following
1 - call /next/10 near the head of the channel
2 - record the last item
3 - call /latest
4 - latest returns an item previous to step 2

This happens infrequently, ~once in 50 attempts.

[chriskessel]

Caught it in the act! Here's the Jersey debug info. You can see "next/100" returns 2 URLS, but "latest" gives back an older result.

Mar 12, 2015 12:54:22 PM com.sun.jersey.api.client.filter.LoggingFilter log
INFO: 21 * Client out-bound request
21 > GET http://hub-v2.svc.staging/channel/FlexFlightHistoryReplicatedHubV1Staging/2015/03/12/19/54/06/151/2872370/next/100
21 > Accept: application/json,text/plain

Mar 12, 2015 12:54:23 PM com.sun.jersey.api.client.filter.LoggingFilter log
INFO: 21 * Client in-bound response
21 < 200
21 < Transfer-Encoding: chunked
21 < Server: Hub/2015-03-10.46
21 < Vary: Accept-Encoding
21 < Date: Thu, 12 Mar 2015 19:54:22 GMT
21 < Content-Type: application/json
21 <
{
"_links" : {
"self" : {
"href" : "http://hub-v2.svc.staging/channel/FlexFlightHistoryReplicatedHubV1Staging/2015/03/12/19/54/06/151/2872370/next/100"
},
"next" : {
"href" : "http://hub-v2.svc.staging/channel/FlexFlightHistoryReplicatedHubV1Staging/2015/03/12/19/54/21/817/2872372/next/100"
},
"previous" : {
"href" : "http://hub-v2.svc.staging/channel/FlexFlightHistoryReplicatedHubV1Staging/2015/03/12/19/54/21/777/2872371/previous/100"
},
"uris" : [ "http://hub-v2.svc.staging/channel/FlexFlightHistoryReplicatedHubV1Staging/2015/03/12/19/54/21/777/2872371", "http://hub-v2.svc.staging/channel/FlexFlightHistoryReplicatedHubV1Staging/2015/03/12/19/54/21/817/2872372" ]
}
}

Mar 12, 2015 12:54:23 PM com.sun.jersey.api.client.filter.LoggingFilter log
INFO: 22 * Client out-bound request
22 > GET http://hub-v2.svc.staging:80/channel/FlexFlightHistoryReplicatedHubV1Staging/latest?stable=true
22 > Accept: application/json,text/plain

Last uri: http://hub-v2.svc.staging/channel/FlexFlightHistoryReplicatedHubV1Staging/2015/03/12/19/54/21/817/2872372
Mar 12, 2015 12:54:23 PM com.sun.jersey.api.client.filter.LoggingFilter log
INFO: 22 * Client in-bound response
22 < 303
22 < Server: Hub/2015-03-10.46
22 < Vary: Accept-Encoding
22 < Content-Length: 0
22 < Date: Thu, 12 Mar 2015 19:54:22 GMT
22 < Location: http://hub-v2.svc.staging/channel/FlexFlightHistoryReplicatedHubV1Staging/2015/03/12/19/54/06/151/2872370
22 <

[moulliet]

Yes, we can also recreate, and are debugging now :)

[moulliet]

Our test against staging data, which could reproduce the issue 5 times per minute, has been running for 5 minutes with no issues.