Not inserting all data into my database from cartodb.com api

[iiVi]

@harimohanraj89 @DrRobotmck @WilliamfShaw @bigtone1284
Wrote a script to grab all parking regulation info from my cartodb.com db via an api call, and create these in my database. However, when I run this, it only grabs about 5k of the roughly 360k objects.

var testAdd = function () {
$.ajax({
url: 'http://andrewcarton1.cartodb.com/api/v2/sql?format=GeoJSON&q=SELECT%20*%20FROM%20parking_regulationshp_1&api_key=a8614e1207568c75211faba281eda4f8d56c8164',
method: 'get'
}).done(testAddData);
};

var testAddData = function (regulationsCarto) {
debugger;
for(var i = 0; i < regulationsCarto.features.length; i++) {
$.ajax({
url: '/regulations/newpoop',
method: 'post',
data: {
object_id: regulationsCarto.features[i].properties.objectid,
borough: regulationsCarto.features[i].properties.sg_key_bor,
location_id: regulationsCarto.features[i].properties.sg_order_n,
sign_sequence: regulationsCarto.features[i].properties.sg_seqno_n,
curb_distance: regulationsCarto.features[i].properties.sr_dist,
sign_code: regulationsCarto.features[i].properties.sg_mutcd_c,
sign_faces: regulationsCarto.features[i].properties.sg_sign_fc,
sign_arrow: regulationsCarto.features[i].properties.sg_arrow_d,
sign_description: regulationsCarto.features[i].properties.signdesc1,
cartodb_id: regulationsCarto.features[i].properties.cartodb_id,
x: regulationsCarto.features[i].geometry.coordinates[1],
y: regulationsCarto.features[i].geometry.coordinates[0]
}
}).done(function() {
console.log('done with one');
});
};
};

[WilliamfShaw]

@iiVi Ok, what have you tried to do to resolve this? Have you checked in on the documentation for the API regarding rate limits? Have you checked to see if there is a bug somewhere in your code preventing it from seeding further?

Also please use backticks when inserting code

var poop = "poop" 

You can do this with the ```

[iiVi]

Docs say nothing about rate limits. Not getting any errors anywhere when running this script. Will seed a number of the objects, but not all.

[WilliamfShaw]

taken

[harimohanraj89]

Left some comments below.

var testAdd = function () {
    $.ajax({
        url: 'http://andrewcarton1.cartodb.com/api/v2/sql?format=GeoJSON&q=SELECT%20*%20FROM%20parking_regulationshp_1&api_key=a8614e1207568c75211faba281eda4f8d56c8164',
        method: 'get'
    }).done(testAddData);
};

var testAddData = function (regulationsCarto) {
        // Good use of a debugger
        // What is the length of `regulationsCarto` here?
        // Is it the full length? If so, then the code below has problems. Else, the code above (with cartodb) has problems
    debugger; 
    for(var i = 0; i < regulationsCarto.features.length; i++) {
        $.ajax({
            url: '/regulations/newpoop',
            method: 'post',
            data: {
                object_id: regulationsCarto.features[i].properties.objectid,
                borough: regulationsCarto.features[i].properties.sg_key_bor,
                location_id: regulationsCarto.features[i].properties.sg_order_n,
                sign_sequence: regulationsCarto.features[i].properties.sg_seqno_n,
                curb_distance: regulationsCarto.features[i].properties.sr_dist,
                sign_code: regulationsCarto.features[i].properties.sg_mutcd_c,
                sign_faces: regulationsCarto.features[i].properties.sg_sign_fc,
                sign_arrow: regulationsCarto.features[i].properties.sg_arrow_d,
                sign_description: regulationsCarto.features[i].properties.signdesc1,
                cartodb_id: regulationsCarto.features[i].properties.cartodb_id,
                x: regulationsCarto.features[i].geometry.coordinates[1],
                y: regulationsCarto.features[i].geometry.coordinates[0]
            }
        }).done(function() {
            console.log('done with one');
        }); // Can you use a .fail() to console log failures?
                    // When you do, do you see the right number failures? (total - successes)
                    // Can you use .fail() to gain more information about WHY things are failing? (What response
                    // do you get from your API?
                    // Can you try with a smaller dataset and identify patterns as to which ones are succeeding
                    // and which ones are failing?
    };
};

[harimohanraj89]

@WilliamfShaw All yours.

[iiVi]

at my debugger, i'm getting the right total number, but cannot immediately see any bugs with the following code. Checking .fail now

[iiVi]

also, not seeing anything other than 200's come back. it just stops at around 5k ish objects posted

[iiVi]

AAHHHHH, THIS IS SO F*CKEDDDDDD