Embedding a large dataset of sentences

[ksrinivs64]

Hi is there anything in particular one needs to do to embed millions of sentences? Do we need to batch it? Right now, am following the code in the python notebook from examples which looks like this:
` input_placeholder = tf.placeholder(tf.string, shape=(None))
encoding_tensor = embed_transform(input_placeholder)

with tf.Session() as session:
    session.run([tf.global_variables_initializer(), tf.tables_initializer()])
    corpus_sentence_embedding = session.run(encoding_tensor, feed_dict={input_placeholder: corpus_sentences})`

But that seems to cause my system to run out of memory for 100K sentences. I'm running this on a laptop with 16G memory but will eventually need to scale to millions of sentences. Thanks!

[vbardiovskyg]

Hi,

yes, the data fed into the graph is kept in memory, so it needs to be batched. If the first step is to just have the final embeddings stored in a file, one can just linearly process the input file in batches, embed the batch, and store into a file.