Pagination on the rackspace DNS api?

[thattommyhall]

If I call

Fog::DNS.new(RACKSPACE_CREDENTIALS).zones.map(&:domain).count
=> 100

Whereas I know I have 317 domains.
I assume the rackspace API does pagination and your provider does not get all the results.

I may have time to fix it myself soon,
Tom Hall

[geemus]

@thattommyhall yeah. There is some weirdness there I fear. I believe .each should "do the right thing" but map doesn't for reasons I'm not totally certain of.

[thattommyhall]

Fog::DNS.new(RACKSPACE).zones.each{ |z| puts z.domain }

Fog::Rackspace::Errors::BadRequest: [HTTP 400] Fog::Rackspace::Errors::BadRequest
    from /var/lib/gems/1.9.1/gems/excon-0.23.0/lib/excon/middlewares/expects.rb:10:in `response_call'
    from /var/lib/gems/1.9.1/gems/excon-0.23.0/lib/excon/connection.rb:353:in `response'
    from /var/lib/gems/1.9.1/gems/excon-0.23.0/lib/excon/connection.rb:247:in `request'
    from /var/lib/gems/1.9.1/gems/fog-1.12.1/lib/fog/core/connection.rb:25:in `request'
    from /var/lib/gems/1.9.1/gems/fog-1.12.1/lib/fog/rackspace/dns.rb:114:in `request'
    from /var/lib/gems/1.9.1/gems/fog-1.12.1/lib/fog/rackspace/requests/dns/list_domains.rb:15:in `list_domains'
    from /var/lib/gems/1.9.1/gems/fog-1.12.1/lib/fog/rackspace/models/dns/zones.rb:37:in `each'
    from (irb):113
    from /usr/bin/irb:12:in `<main>'

At the 100 mark it gives out

[thattommyhall]

irb(main):114:0> Fog::DNS.new(RACKSPACE).zones.count
=> 100

I'm confused

[geemus]

@thattommyhall - I'm confused too, I fear.

@krames - could you help us figure out whats up? Thanks!

[krames]

@thattommyhall Sorry about that! Let me see if I can't dig into the issue and figure out what's going on for you.

[krames]

@thattommyhall I am currently looking into this. I think there might be a bug in there. Let you know!

[krames]

@geemus @thattommyhall In researching this I noticed something interesting. The zones.rb class seems to make a couple extraneous calls to DNS.

When you execute the following:

zones.each {|z| z.domain}

The code actually lazy initializes itself by calling the all method. This queries DNS for the first page of DNS records (100 records). After it does this, the each method then calls DNS again for each page (possibly more than that, I think there is a bug).

It would seem to me that the all method should return the complete list of DNS records (317 in Tommy's case) and then the each should iterate over a collection like normal. Then we should have a method to allow developers to process this information a page at a time.

Based on that assumption, I created the following gist -> https://gist.github.com/krames/5800459

This this the best approach? How should this work in fog?

Thanks!

[geemus]

@krames - all probably should not load everything, followed by iteration, I fear. If a user has a really large list (say 10,000) loading/iterating over it all becomes pretty untenable. What we have done else where (ie S3) is to have all get a page and each do the iteration and pass to the block one page at a time. That way you can choose between the behaviors. If we made all just return everything, how would you just get a single page?

[krames]

I have submitted a pull request to address this issue.