S3 AWSPager for ListObjects bug

[nhibberd]

This is the old code that works (1.0.1)

instance AWSPager ListObjects where
        page rq rs
          | stop (rs ^. lorsIsTruncated) = Nothing
          | isNothing
              (rs ^.
                 choice (^. lorsNextMarker)
                   (^? (lorsContents . _last . oKey)))
            = Nothing
          | otherwise =
            Just $ rq &
              loMarker .~
                rs ^.
                  choice (^. lorsNextMarker)
                    (^? (lorsContents . _last . oKey))

This version (1.3.0) does not

instance AWSPager ListObjects where
        page rq rs
          | stop
              (rs ^.
                 choice (^. lorsNextMarker)
                   (^? (lorsContents . _last . oKey)))
            = Nothing
          | stop (rs ^. lorsContents) = Nothing
          | otherwise =
            Just $ rq &
              loMarker .~
                rs ^.
                  choice (^. lorsNextMarker)
                    (^? (lorsContents . _last . oKey))

The situation we see is that we have the first page return with a set of contents and a set of common prefixes. This page is the complete result we expect. NextMarker is Nothing as expected, truncated is set to Just False. However the pager requests a new page because the contents is non empty (the new clause in the stop condition above + the missing truncated check). The generated code looks slightly off anyway because the second condition totally overlaps with the first by the looks of it anyway. The end result of this is that after calling paginate we end up with all the common prefixes returned twice (from the first page, and then the incorrect second page). Our experience with this API is the isTruncated is the only thing that can be trusted - nothing should be relying on the existence (or not) of next marker or contents.

[brendanhay]

Looks like fbc25dd broke it, which was designed to support a slightly newer pagination definition.

I can't directly roll back, since the pagination format has changed, so I'll see if I can get a fix out shortly.

[brendanhay]

The diff between 1.0.1 and the fixes in #219 show it should be back to working as expected. I'll let the tests go through and release a patch to Hackage with the fix from #217.