issue about data preprocess

[shangqing-liu]

Hi,
May I ask when I read your codes about reader.py, I doubted the function process_dataset, the sample of the maximum contexts:

safe_limit = tf.cast(tf.maximum(num_contexts_per_example, self.config.MAX_CONTEXTS), tf.int32)
rand_indices = tf.random_shuffle(tf.range(safe_limit))[:self.config.MAX_CONTEXTS]
contexts = tf.gather(all_contexts, rand_indices) # (max_contexts,)

seems will be array out of bounds.

[urialon]

Hi,
There is no array out of bound, because there is padding. Even when there are less than max_contexts, there are paddings instead.

Best,
Uri

[shangqing-liu]

got it, thanks very much @urialon