adds batch jobs for long-running geometry creation

[andy-esch]

For larger tables written to carto with the lnglat flag, the geometry creation step times out. To get around this, we can send the job to the SQL Batch API. The upside is that there will not be a timeout, the downside is that it has to sit in a queue.

The new flow for cc.write is that if the dataframe has more than 1e5 rows, the geometry creation query will be sent to the Batch SQL API and a BatchJobStatus object will be returned. This object can be polled with the .status() method. If the number of rows is at or below 1e5, the normal SQL API will be used for geometry creation and cc.write will return None.

Example usage:

import time
job = cc.write(df, 'new_table',
               lnglat=('lng_col', 'lat_col'))
while job is not None:
    curr_status = job.status()['status']
    if curr_status in ('done', 'failed', 'canceled', 'unknown', ):
        print(curr_status)
        break
    time.sleep(5)

ToDos

• turn the pandas.groupby into a generator
• ensure that there's test coverage to all the new pieces
  • BatchJobStatus
  • lnglat for batch jobs

[coveralls]

Coverage decreased (-1.05%) to 89.474% when pulling 6b9a1a8 on batch-lnglat-update into 17d71bb on master.

[coveralls]

Coverage increased (+0.06%) to 90.582% when pulling ecc8cd0 on batch-lnglat-update into 17d71bb on master.

[coveralls]

Coverage increased (+0.2%) to 90.72% when pulling fb81885 on batch-lnglat-update into 17d71bb on master.

[coveralls]

Coverage increased (+0.2%) to 90.72% when pulling fb81885 on batch-lnglat-update into 17d71bb on master.

[coveralls]

Coverage increased (+0.2%) to 90.72% when pulling fb81885 on batch-lnglat-update into 17d71bb on master.

[andy-esch]

@dmed256, could you take a look at this?

[dmed256]

I think it should be:
if df.shape[0] > MAX_IMPORT_ROWS:

[dmed256]

To make sure the / are there:

os.path.join(base_url, 'dataset', table_name)

[dmed256]

Maybe it's ok the way you have it, it's in other places in the code

[andy-esch]

Might as well be careful

[dmed256]

n_rows = context.MAX_IMPORT_ROWS + 1

[andy-esch]

@dmed256 you suggested in #85 a method for returning dataframes from longer running queries. Here we could have a flag or another method for returning the lnglat'd version of the table/dataframe. Soon, I envision us adding more than the lnglat flag for adding geometries.. e.g., street-level geocoding, routing, etc.

But here, we could have a simple new method (or flag to status), which indicates that when status=done, do cc.read('tablename') to return the carto'd dataframe.

What do you think? Too soon?

[andy-esch]

I think too soon. I want to scope it out more before including anything else here.

[coveralls]

Coverage increased (+0.2%) to 90.733% when pulling 75ecd0b on batch-lnglat-update into 17d71bb on master.

[coveralls]

Coverage increased (+0.2%) to 90.733% when pulling fa2d5b6 on batch-lnglat-update into 17d71bb on master.

[coveralls]

Coverage increased (+0.2%) to 90.733% when pulling fa2d5b6 on batch-lnglat-update into 17d71bb on master.

[andy-esch]

To avoid the issues discussed in #153 and #210, I've done some refactoring to investigate more efficient ways of getting the tables to (1) display in the dashboard and get a dataset page without being a ghost table, and (2) avoid timeouts by finding cleverer ways of populating the geometry field by avoiding UPDATE statements and offloading the queries to the Batch SQL API.

Some timings without the Batch SQL API for the entire write process:

[coveralls]

Changes Unknown when pulling 33171e0 on batch-lnglat-update into ** on master**.