Allow batching subscription deletions

Hey, I am trying to unsubscribe from multiple topics:

    subscription.forEach(function (sub) {
      sub.delete(function (error) {
        //code
      });
    });

Everytime I try this, i get an unhandled error:

ApiError: The service was unable to fulfill your request. Please try again. [code=8a75]
at new util.ApiError (/Users/X/Y/node_modules/gcloud/lib/common/util.js:92:10)
    at Object.parseHttpRespBody (/Users/X/Y/node_modules/gcloud/lib/common/util.js:170:30)
    at Object.handleResp (/Users/X/Y/node_modules/gcloud/lib/common/util.js:110:18)
    at /Users/X/Y/node_modules/gcloud/lib/common/util.js:422:12
    at Request.onResponse [as _callback] (/Users/X/Y/node_modules/retry-request/index.js:106:7)
    at Request.self.callback (/Users/X/Y/node_modules/gcloud/node_modules/request/request.js:198:22)
    at emitTwo (events.js:87:13)
    at Request.emit (events.js:172:7)
    at Request.<anonymous> (/Users/X/Y/node_modules/gcloud/node_modules/request/request.js:1035:10)
    at emitOne (events.js:82:20)

Any idea?

EDIT: interesting enough, even though it throws an error, it still seems to delete the subscriptions.

[stephenplusplus]

Thanks for reporting this, @Moezalez. I'll pull in the Pub/Sub expert for this issue: @tmatsuo does this sound like a rate limit error?

Our library will automatically back off, then retry requests after it receives a response from the API that indicates the server needs a break. However, it will only retry up until 3 failed requests.

Edit: We do have an option to increase this limit, but it's currently broken :(

Thanks, @stephenplusplus.
It seems indeed to have something to do with rate limiting.
I found out that giving the mechanism more time evades the problems:

    (function unsubscriptionLoop (i) {
      setTimeout(function () {
        subscriptions[i].delete(function (error) {
          //code
        });
        if (i > 0) {
          i--;
          unsubscriptionLoop(i);
        }
      }, 200);
    })(subscriptions.length - 1);

But that is obviously not an ideal solution.

[stephenplusplus]

The ideal solution would be an upstream batch API endpoint that can handle multiple subscriptions being sent to it. @tmatsuo -- any hope of such a thing?

In our library, we could add pubsub.deleteSubscriptions(subscriptions, callback), which would throttle the requests to try to step around rate limits.

// @callmehiphop for thoughts

[callmehiphop]

I agree that we would ideally solve this with a batch API endpoint, however if that's not an option I also think a deleteSubscriptions method with a throttle option would fit well and be consistent with our current APIs.

The only other thing I could think of would be adding throttling logic to all our APIs at the ServiceObject level, letting the user configure the throttle rate at service instantiation time and managing it behind the scenes. This would probably be a semi-large effort aaand maybe impractical.

[stephenplusplus]

The only other thing I could think of would be adding throttling logic to all our APIs at the ServiceObject level, letting the user configure the throttle rate at service instantiation time and managing it behind the scenes. This would probably be a semi-large effort aaand maybe impractical.

Yeah, this would be nice, but I think for the general use case, we're safe with our current logic.

But you make a great point... this is just one part of our API. We have many delete operations across our service objects. I'm starting to think I would rather let the npm ecosystem solve this for those who need to be able to rapidly delete objects. They can choose from a variety of tools to help them throttle their requests, of course async being one of them:

var async = require('async');
var PARALLEL_LIMIT = 10;

function deleteSubscription(subscription, callback) {
  subscription.delete(callback);
}

async.eachLimit(subscriptions, PARALLEL_LIMIT, deleteSubscription, function(err) {
  if (!err) {
    // Subscriptions deleted.
  }
});

@Moezalez most importantly, what do you think?

[callmehiphop]

I'm starting to think I would rather let the npm ecosystem solve this for those who need to be able to rapidly delete objects. They can choose from a variety of tools to help them throttle their requests

I like this, I think we should document use cases like this, suggesting the use of third party libraries.

[tmatsuo]

Batch deletion is currently not suported. Maybe you can file a feature request at https://code.google.com/p/cloud-pubsub/issues/list ?

[stephenplusplus]

Oh, nice! Do you happen to have links for the other services, i.e. Datastore, Storage, etc?

[stephenplusplus]

Submitted: https://code.google.com/p/cloud-pubsub/issues/detail?id=28&thanks=28&ts=1454955900

@stephenplusplus You're proposed solution utilizing async seemed to have worked quite well in my case