7. Lessons Learned and Best Practices

Lessons Learned

1. Kaggle, ML/AI blogs, papers and even social media are an incredibly powerful source for state-of-the-art methodologies. It is important to understand the speed at what these methods are evolving, and more stationary sources (such as books) may not fully capture what current practitioners and enthusiasts are working on

2. Trial and error for novel approaches in deep learning (sentiment extraction in this case) will be always part of an exploratory exercise (involving recurrent experiments). Be ready to get frustrated at times

3. At a certain point, the models stopped working with the latest transformer’s library version, so we had to go back to the old version as per the creators of simple transformers. The error is still being corrected. The downside of using such libraries compared to the well-established ones (such as Scikit learn) is that only a limited amount of people are working on it in the backend, and errors are not common to solve.

4. The JSON format did not work well when fitting the data to the models (although that was the main format in the training material provided online). Using a list of dictionaries worked best for us as provided in the setup code.

5. Some models did not show the performance we expected. For instance, XlM-based architectures were challenging to run, especially the larger versions with several layers and parameters. It also needs a lot of space to store the models (2.5GB per epoch). A reason to its low performance compared to other alternative models os that XLM is not a bidirectional model. It is an autoregressive model that only processes the text in one direction (i.e. based on every word before the target word to predict the target word). Generally speaking, models are good at predicting the next word in the sentence. However, question-answering tasks require an understanding of the full context, hence this model could not handle it properly.

6. TODO--> Bert models are autoencoding models that process all the context simultaneously which can perform well in question answering tasks. However, They can’t do language modeling such as predicting the next word of the sentence well.

Best Practices

1. The initial setup can take time to complete, especially when working in teams. Make sure to align on common aspects such as packages, use of GPU and platforms for model design and version control.

2. Keep track of every experiment, include details (e.g. hyperparameters), and share it with teammates to avoid duplication of work. In our case, we used GitHub Projects (Kanban framework) to list the different experiments that were performed, as well as their corresponding hyperparameters and Jaccard Score.

3. Documentation is very important for multiple reasons: it provides a walk-through for all kinds of users (technical and non-technical), it provides more clarity to the developer (“is this line of code doing what I’m including in the comment?”), and it makes it available for future use/reference.

4. Running these types of models requires a lot of computational power. Cloud-based alternatives can help addressing this challenge. In our case, we used Google Colab as it offers free GPU (critical for deep learning).