This repository is meant to help recreate the results from the paper: Problems with Cosine as a Measure of Embedding Similarity for High Frequency Words (https://arxiv.org/abs/2205.05092). The core analysis for the two papers are in WiC_analysis.ipynb
, scws_analysis.ipynb
, and Section3/wiki_bounding_balls_analysis.ipynb
. In order to reproduce the results, there are five main steps: downloading various data sources (Wikipedia, BookCorpus, WiCs, SCWS), calculating word frequencies from Wikipedia and BookCorpus, querying for keywords from Wikipedia, building contexutalized embeddings, performing analysis on these embeddings.
In more detail:
-
Gather a large corpus of text to create contextualized word embeddings Download a large corpus of text which will be used to create contextualized word embeddings. We used the March 1, 2020 Wikimedia Download which can be found here: https://dumps.wikimedia.org/. We later want to be able to query this corpus for examples of words in context. We scramble this Wikipedia corpus (at the sentence level) so that we can query the first 50 examples found and avoid biases certain Wikipedia pages/topics.
-
Calculate word frequencies for Wikipedia and BookCorpus We used tools such as wikiextractor and wiki-word-frequencies to help calculate the word frequencies of Wikipedia corpus we have. We then used word frequencies from Complex Word Identification (CWI) Shared Task 2018 project as a way to estimate BookCorpus word frequencies(https://github.com/nathanshartmann/NILC-at-CWI-2018). We combined both frequencies
calculate_bert_word_frequencies.ipynb
and the resulting filebert_word_frequencies.csv
. -
Preprocess WiC, SCWS data Download the train and dev datsets for WiC (place in a folder called
WiC_data
. Preprocess datapre_processing_wic.ipynb
. Download the train and dev datsets for SCWS (place in a folder calledSCWS_data
). Preprocess data using the file:scws_embeddings.ipynb
-
Create contextualized word embeddings for sentences for the WiC dataset This involves embedding the entire sentence and then extract the word embedding associated with the word of interest. We then used Hugging Face's library to create contexutalized word embeddings. See file:
utils/create_embeddings/creating_word_embeddings.py
. Analyze the resultsWiC_analysis.ipynb
andscws_analysis.ipynb
-
Preprocess Wiki data 30K Random Word Sample We next selected 30,000 words (across a variety of word frequencies buckets) as our "words of interest". We then query for examples of these words in context in our Wikipedia corpus
find_words_in_wikipedia.ipynb
. Retain all words with at least 50 examples. -
Create contextualized word embeddings for random pairs of sentences from Wikipedia corpus. We next selected 30,000 words (across a variety of word frequencies buckets) as our "words of interest". We then queried our Wikipedia corpus for the sentences of these 30,000 words
find_words_in_wikipedia.ipynb
. -
WiC Word Sample We then select all the words from the WiC task as "words of interest" and query for examples of these words in context in our Wikipedia corpus. Retain all words with at least 50 examples.
-
Create contexutalized word embeddings for the Wikiwords Repeat step 4 to create contextualized word embeddings for the 30K Random Word Sample and the WiC Word Sample.
-
Calculate the geometric space occupied by contextualized word embeddings of the same word Analyze the geometric space of the random Wikipedia words and the WiC sample words. See files:
Section3/creating_word_embedding_bounding_balls.py
,Section3/measure_bounding_balls.ipynb
,Section3/wiki_bounding_balls_analysis.ipynb
-
To how these results compare to other measurements see: Measuring the geometric space of contextual embeeding using other metrics such as minimum bounding hyperspheres, norms, and euclidean distance from the centroid. See files:
Section_3/other_measurements/pre_processing_other_measurements.ipynb
andSection_3/other_measurements/other_measurements.ipynb