Batch Request For Embeds

[OrenLeung]

What it does

• Add non-parallel Batch Embeds
• Parallel Batch Embeds through python Threading Workers or concurrent.futures.ThreadPoolExecutor (still researching to figure out which is better)

Design Decision

• chose thread executor over multi processing executor as it is IO-bound and not CPU-bound so most of the time the threads are waiting for a network response
• the response must be in order (test added)

Future PR

• add default num_workers depend on user's # of cores add default num_workers depend on user's # of cores #46
• exponential decay retrying exponential decay retrying #47

Tests

• added unit tests for multiple batches
• test that it is in order

Basic Performance Test for 25k texts

• @jalammar reported it takes 5 hours for non-parallel

• batching resulted in 3mins & 30 secs

letters = string.ascii_lowercase

def random_word():
    return ''.join(random.choice(letters) for i in range(10))

def random_sentence(num):
    sentence = ""

    for i in range(num):
        sentence += random_word()

    return sentence

def list_of_list(num):
    arr = []

    for i in range(num):
        arr.append(random_sentence(50))
    
    return arr

co = cohere.Client(
    API_KEY)

prediction = co.embed(
    model='small',
    texts=list_of_list(25000))

print(len(prediction.embeddings))

[jimwu6]

LGTM