BigTable: Async client for high throughput mutate_rows

[mayurjain0312]

Ubuntu 16.04
Python version and virtual environment information python --version - python 2.7.12

With a batch size of 300, total 3 nodes in an instance, the write throughput is not good for BigTable by using mutate_rows api call.

BULK_WRITE_BATCH_SIZE = 300
with open(file_path) as sensor_data_input_file:
    list_direct_row_obj = []
    for line in sensor_data_input_file:
        if not line.strip():
            continue

        sensor_json_data = json.loads(line)
        row_key = create_sensor_data_id(sensor_json_data)
        value = line

        direct_row_obj = bigtable.row.DirectRow(row_key, table)

        column_id = 'column_id_data'.encode('utf-8')
        direct_row_obj.set_cell(column_family_id, column_id, value.encode('utf-8'))


        list_direct_row_obj.append(direct_row_obj)
        direct_row_obj = None

        if len(list_direct_row_obj) == BULK_WRITE_BATCH_SIZE:
            table.mutate_rows(list_direct_row_obj)
            list_direct_row_obj[:] = []

    if list_direct_row_obj:
        table.mutate_rows(list_direct_row_obj)

[sduskis]

Can you please describe some of the symptoms you're experiencing? table.mutate_rows is a synchronous operation, so you're limited to one call at a time. How many rows per second are you seeing on the cloud console?

[mayurjain0312]

Is this the only "bulk write" option that BigTable offers?
From the console, I can see around 800 rows written per second.
(Production instance with 3 nodes)

[mayurjain0312]

We are currently using mongodb as our no-sql database and we would prefer moving to BigTable. As far as the speed of writing data to database is concerned, I see that mongodb is far exceeding BigTable(unless I am not using BigTable in the right manner)

[mayurjain0312]

@sduskis: any updates?

[sduskis]

A 3 node Cloud Bigtable cluster can handle writing at least 30K rows per second, and each additional node adds 10K rows per second. We tested up to 3,500 nodes with Cloud Bigtable with a linear increase in performance.

The 800 rows per second is a limitation of the python client and what ever VM you're on. The process sends 300 rows, and waits for a response. You can run multiple processes that each read in a different file to improve total throughput against your cluster.

The Java client has an async client that performs far better in these situations, since it has a robust async client. We have not ported that functionality to python yet. Here is an example with the Java version.

(FWIW, I'm the primary developer on the Java client)

[mayurjain0312]

@sduskis : Thanks for responding.
Do you know when can this async functionality for Python be ready?

[sduskis]

We don't have an ETA for this functionality in Python.

[basphilippus]

Hi,

Is there any progress on this feature? I would really like to scale my writes to Bigtable with Python code.

[sduskis]

We still do not have concrete plans to implement this feature.