Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support batching in 1.6 driver #55

Closed
bigbes opened this issue Apr 3, 2015 · 4 comments
Closed

Support batching in 1.6 driver #55

bigbes opened this issue Apr 3, 2015 · 4 comments
Labels
feature A new functionality

Comments

@bigbes
Copy link
Contributor

bigbes commented Apr 3, 2015

No description provided.

@bigbes bigbes self-assigned this Apr 8, 2015
@buriy
Copy link

buriy commented Apr 19, 2016

would like to have batching support in the client.

@ghost
Copy link

ghost commented Oct 20, 2016

Yes, something to reduce round-trips from client to database would certainly be a great feature.

@bigbes bigbes added this to the 0.6.2 milestone May 23, 2017
@bigbes bigbes added the feature A new functionality label May 23, 2017
@bofm
Copy link

bofm commented Dec 4, 2017

A dirty workaround might help for now. Here is a batch insert variant.

from itertools import islice

def load_tarantool(t, items, space_name):
    space_sid = t.schema.get_space(space_name).sid
    items = iter(items)
    chunk_size = 1000
    chunk = tuple(islice(items, chunk_size))
    n = 0
    while chunk:
        payload = b''.join(bytes(RequestInsert(t, space_sid, item)) for item in chunk)
        t._socket.sendall(payload)
        try:
            assert all(Response(t, t._read_response()) for _ in chunk)
        except SchemaReloadException as e:
            t.update_schema(e.schema_version)
            continue
        n += len(chunk)
        chunk = tuple(islice(items, chunk_size))
    print(f'inserted {n} items') 

@DifferentialOrange
Copy link
Member

There is no any native support for batching in basic IPROTO requests. Just sending a sequence of inserts isn't nice since some of them may fail in the middle and we won't have any possibility to handle it properly.

On the other hand, crud module has *_many methods which is essentially a batch insert/replace/upsert for sharded systems. These handles will be supported as part of the #205 issue. So I'm closing this one in favor of #205

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature A new functionality
Projects
None yet
Development

No branches or pull requests

5 participants