Generators file for low-memory users - Code Provided

[thethiny]

Hi,
I would suggest these 2 generators for usage with people with small memory, or for GPUs with low memory.

def _generator(sentence_pairs, batch_size, seq_len):
    while True:
        inp, outp = gen_batch_inputs(sentence_pairs, token_dict, token_list, seq_len=seq_len)
        for i in range (0, len(sentence_pairs), batch_size):
            yield [inp[x][i:i+batch_size] for x in range(len(inp))], [outp[x][i:i+batch_size] for x in range (len(outp))]
            
def _generator_v(sentence_pairs, batch_size, seq_len):
    while True:
        for i in range(0, len(sentence_pairs), batch_size):
            yield gen_batch_inputs(sentence_pairs[i:i+batch_size], token_dict, token_list, seq_len=seq_len)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.