add a new PyText get_num_examples_from_batch function in model

Summary:
Usually in PyText dataloader, each row (sentence) is read as an example, i.e. create List[sentences], and the number of examples in the batch is simply len(List[data]).

This is not correct for language model: in LM, each word is considered as an example.

Reviewed By: kmalik22

Differential Revision: D21032297

fbshipit-source-id: 619e3509e2dadb64431700516e23c4911cc4699a

pytext/models/language_models/lmlstm.py

@@ -142,6 +142,11 @@ def arrange_targets(self, tensor_dict):
         tokens, seq_lens, _ = tensor_dict["tokens"]
         return (tokens[:, 1:].contiguous(), seq_lens - 1)
 
+    def get_num_examples_from_batch(self, batch):
+        targets = self.arrange_targets(batch)
+        num_words_in_batch = targets[1].sum().item()
+        return num_words_in_batch
+
     def get_export_input_names(self, tensorizers):
         return ["tokens_vals", "tokens_lens"]
 

pytext/models/model.py

@@ -228,6 +228,17 @@ def arrange_caffe2_model_inputs(self, tensor_dict):
                 flat_model_inputs.append(model_input)
         return flat_model_inputs
 
+    def get_num_examples_from_batch(self, batch):
+        """
+        usually, the number of examples in the batch is just number
+        of rows (len) in the batch, but this may not be true. for
+        example, in language model, each row has multiple words, and
+        the total number of examples is the total number of words across
+        all rows in the batch. Thus LM model needs to override this function.
+        """
+        targets = self.arrange_targets(batch)
+        return len(targets)
+
 
 class Model(BaseModel):
     """