With the significant developments in the field of Artificial Intelligence (AI), complex models with higher performance have been developed. One of these models is the Transformer structured BERT model. It has been observed that BERT-based sentiment analysis models are more successful than sentiment analysis models built using lexicon-based (LB) and Machine learning (ML) approaches. In this research, we develop a local approach model-agnostic explainability method for the explainability of BERT-based sentiment analysis models working on a Turkish database. Adjectives, adverbs and predicates in sentences are identified and replaced with their antonyms, and the effect of these word types on the final prediction of the model is observed.
The Turkish text given as input to the system is directly given as input to the BERT-based sentiment analysis model. The prediction result and the classification score generated by the model for the Turkish text are recorded. In order to understand which words contribute to the model's prediction, we need to find all critical words (adjectives, adverbs and verbs) in the text. Therefore, the sentence is first parsed into Part of Speech (POS) tags. This identifies the linguistic type of each word in the sentence. Then, for each critical word identified in the text, the following steps are taken:
The POS tag is used to check whether the word is an adjective or an adverb. If it is an adjective or an adverb, its antonym is extracted from the previously created Adj/Adv-AntonymDict dictionary. If the word's antonym is not in the dictionary, the word's antonym is extracted using the WordNet API. The word in the text is replaced by the antonym extracted from the dictionary. The newly created text is fed as input to the BERT model and the model's prediction for the new text is saved. The contribution of the word to the model prediction is found using the word contribution equation. If the word type is not an adjective or adverb, this step is skipped.
The word type of the word is checked by the dependency parser to see if it is predicate. If it is predicate and affirmative, its negative is extracted, if negative, its affirmative is extracted from the Verb-AntonymDict dictionary. If there is no positive/negative equivalent of the predicate in the dictionary, the error "Predicate Not Found in Dictionary" is returned. In the text, the word is replaced by the positive/negative predicate extracted from the dictionary. The newly created text is given as input to the BERT model. The model's prediction for the new text is recorded. The contribution of the word to the model prediction is found using the word contribution equation. If the word type is not verb, this step is skipped.
Once the contribution of each adjective, adverb and verb in the text to the BERT model score has been calculated, the words are ranked from most to least contributory to the model decision and coloured according to their contribution score.
The system uses 2 different sentiment analysis models based on BERT. These are The model trained by Savaş Yıldırım for sentiment analysis of Turkish texts and The model trained by Gürkan Şahin for sentiment classification of Turkish product reviews.
The training parameters of the two models are as follows
- maximum sequence length = 128
- per gpu train batch size = 32
- learning rate = 0.00002
- number of training epochs = 3.0
Onur Yılmaz's POS tagging tool and the TurkishMorphology module of the Zemberek python library have been used to detect adjectives and adverbs in Turkish texts given as input to the system. The antonyms of each detected word type are accessed via Turkish WordNet and Lexicala Web API. In addition, Adj/Adv-AntonymDict dictionary was manually created by manually adding antonym pairs to the Turkish WordNet and Lexicala Web API results. In addition, a Verb-AntonymDict dictionary consisting of positive/negative verb pairs was created by considering the conjugation of verbs and accusatives found in the test comments. These dictionaries are used to define antonyms in the system.
In the repository you can find research codes and research article.