HTTP Error 413 Request Entity Too Large

[jrobertsvml]

Any way to fix this?

Error: [Guzzle\Http\Exception\ClientErrorResponseException] Client error response
[status code] 413
[reason phrase] Request Entity Too Large
[url] https://dns.api.rackspacecloud.com/v1.0/432716/domains/3903288

[jamiehannaford]

@jrobertsvml Hi Joseph, can you paste some of the code you're using which results in the 413 error?

[jrobertsvml]

public function syncRackspaceDns($client = null) {
        if(is_null($client)){
            throw new NotFoundException('Invalid Rackspace client');
        }

        $service = $client->dnsService();
        $domains = $service->domainList(true);

        foreach ($domains As $d) {
            // Check for matching asset, if none exists, create it
            $asset = $this->Assets->find()
                ->where(['Assets.fqdn' => $d->name])
                ->first();

            if (is_null($asset)) {  // Create new asset
                $asset = $this->Assets->newEntity();

                $asset->fqdn = $d->name;

                if (!$this->Assets->save($asset, ['validate' => false]))
                    die(__('Could not create new asset for: '.$d->name));
            }

            // Test for existence
            $myDomain = $this->Domains->find('all', [
                'conditions' => ['rackspace_id' => $d->id]
            ])->first();

            if (is_null($myDomain)) {
                $newDomain = $this->Domains->newEntity();
            } else {
                $newDomain = $this->Domains->get($myDomain->id);
            }

            // Set attributes
            $newDomain->asset_id    = $asset->id;
            $newDomain->rackspace_id= $d->id;
            $newDomain->created     = $d->created;
            $newDomain->modified    = $d->updated;
            $newDomain->fqdn        = $d->name;
            $newDomain->ttl         = $d->ttl;
            $newDomain->comment     = $d->comment;
            $newDomain->email_address = $d->emailAddress;
            $newDomain->domain_name_provider_id = 4; // Rackspace

            if (!$this->Domains->save($newDomain, ['validate' => false]))
                die(__('Could not save domain: '.$d->name));

            // Get all records for this domain
            $thisDomain = $service->domain($d->id);

            $records = $thisDomain->recordList();
            foreach ($records as $r) {
                // Test for existence
                $myRecord = $this->DomainRecords->find('all', [
                    'conditions' => ['domain_id' => $newDomain->id]
                ])->first();

                if (is_null($myRecord)) {
                    $newRecord = $this->DomainRecords->newEntity();
                } else {
                    $newRecord = $this->DomainRecords->get($myRecord->id);
                }

                $newRecord->domain_id           = $newDomain->id;
                $newRecord->rackspace_id        = $r->id;
                $newRecord->hostname            = $r->name;
                $newRecord->target_ip_address   = $r->data;
                $newRecord->time_to_live        = $r->ttl;
                $newRecord->created             = $r->created;
                $newRecord->modified            = $r->updated;

                if ($r->type == 'A')
                    $newRecord->domain_record_type_id = 1;
                elseif ($r->type == 'CNAME')
                    $newRecord->domain_record_type_id = 2;
                elseif ($r->type == 'MX')
                    $newRecord->domain_record_type_id = 3;
                elseif ($r->type == 'NS')
                    $newRecord->domain_record_type_id = 4;
                elseif ($r->type == 'SRV')
                    $newRecord->domain_record_type_id = 5;
                elseif ($r->type == 'TXT')
                    $newRecord->domain_record_type_id = 6;

                if (!$this->DomainRecords->save($newRecord, ['validate' => false])) {
                    die(__('Could not save domain record: '.$r->name));
                }

                $this->out($newRecord->hostname . ' => ' . $newRecord->target_ip_address . ' TTL ' . $newRecord->time_to_live);
            }
        }
    }

This line is where the stack trace begins:

// Get all records for this domain
$thisDomain = $service->domain($d->id);

[jamiehannaford]

@jrobertsvml In order to debug further, could you follow a few steps:

1. Install Guzzle with composer require guzzle/guzzle. This will install the Log plugin (allowing you to capture all the HTTP transactions between your client and the remote server).
2. Once this is installed, add this to your code (this will output all HTTP transactions to STDOUT, so make sure it's in some kind of dev environment). If you could e-mail the content to jamie.hannaford@rackspace.com I'll investigate further. Thanks!

[jamiehannaford]

@jrobertsvml So it looks like you're exceeding your HTTP requests quota. The API is basically saying that you're hitting it too many times per minute. If you look at the docs, it says that for GET requests on domain resources, the limit is 60 per minute. If you exceed that, a 413 error will be returned. Here are some ideas that might get around this problem:

1. Contact our support team via a ticket on the Control Panel, or your account manager, and ask if the DNS team can increase your API quota so that your app doesn't run into this issue. You might need to benchmark how many times it's hitting the API each minute. If you don't feel comfortable doing this, I can reach out on your behalf (you'll need to email me your account number).
2. Set up a retry subscriber to automatically retry failed requests. In order to do this, you first need to install Guzzle (composer require guzzle/guzzle) and then add this script to your code. It will retry failed HTTP requests that result in certain HTTP status codes and cURL error codes. But since the DNS quota is defined per minute, I'm not sure how effective this will be.
3. The final approach is to refactor your app to use fewer HTTP requests. You might not need to be making so many per minute; if you do, you might be able to cache them in a DB or on the filesystem.

I hope this helps, and sorry for your inconvenience. If there's anything else I can help with, feel free to comment here.

[jrobertsvml]

I appreciate you looking into the issue and identifying the problem. I'm thinking about refactoring, but is there a way to grab all of the dns entries and their child records all at once?

[jamiehannaford]

You can grab up to 100 domains per request, so it depends on the total size of your domain collection.

In terms of the child records, you will unfortunately need to execute at least 1 request per domain. This is because a record is associated with a domain in a REST hierarchy, so there is no current way to do a global search 😞 Sorry I can't be of more help. If you need any functionality added or altered to the SDK that you think might benefit you and other users, please contact me and I'll see if I can help.

[jrobertsvml]

I'd love to have an API call that returns json containing of all my domains and domain records. I can parse that on my end and only hit the API once.

[jamiehannaford]

@jrobertsvml I've posted your idea on our Product Feedback site, so hopefully they'll be able to look into this feature request in the future. Thanks for the feedback! If there's anything else I can help with, please comment here - otherwise feel free to close this issue.

[jamiehannaford]

@jrobertsvml Is there anything else you need help with? Otherwise, feel free to close this issue :)