6. Key Insights
-
SimpleTransformers is a powerful tool that can be applied to numerous tasks (refer to https://simpletransformers.ai/) without requiring working knowledge in Pytorch or Tensorflow. However, it might not provide as much flexibility, especially when adding customized layers to pre-built models to increase its performance.
-
QuestionAnswering model for sentiment extraction--> This can be considered a novel adaptation to the traditional text extraction via QA. Transformers are so effective and powerful that they can output a text extraction based on a word that was not present in the context (document).
-
Transformers that ran for one epoch can be as effective as those ran for multiple ones (with some exceptions such as XLM models). In fact, some models experienced slightly less predictive power once we reached the third epoch. A possible reason for this to happen is that the model is overfitting the training set; in other words, it is memorizing it (including the noise that will not help generalize for future cases). Once the model assesses the evaluation set (unseen data), its performance decreases
-
Hyperparameter tuning did not dramatically change the results of the model. There is no clear way of determining which parameters are important to tune. (working with the simple transformers models) it seems that they have already tuned the models to perform a certain way.
- We noticed that the quality of the dataset is not fully consistent: some labels seemed subjective compared to others, which could be an issue when training with a transformer model; hence the need for quality data when training any algorithm. Furthermore, we are dealing with sentiment-based labels – it is challenging to "hack" human behavior (but we tried!).
