Apllying code in my own dataset

[nassera2014]

Hi, Thank you for this great job, i'm beginner in BERT, and i want to use it in topic modeling (extraction topics from arabic text), do you have an idea how can i do this using your code? thank you so much.

BR

[vinid]

Hi @nassera2014!

I, unfortunately, do not know much about Arabic, so I am basing my experience on other languages that can be tokenized through the use of the whitespace char. I am assuming you have your documents in a text file arabic_documents.txt, one document per line.

Preprocessing

First thing first, you need to preprocess the documents.

What we usually do is use our preprocessing pipeline (that will remove stopwords and just keep the most frequent 2000 tokens in the vocabulary) or you can run preprocessing in the way you prefer. However, I am not sure how words are tokenized in Arabic, and we currently tokenized based on whitespace. Arabic might require more steps.

If whitespace tokenization is ok for Arabic, this snippet should be a good starting point:

from contextualized_topic_models.utils.preprocessing import SimplePreprocessing

documents = [line.strip() for line in open("arabic_documents.txt").readlines()]
sp = SimplePreprocessing(documents, stopwords_language = "arabic")
preprocessed_documents, unpreprocessed_documents, vocab = sp.preprocess()

Training

Then you can use the rest of our code to run the topic model. For this step, you need:

1. the documents we have prepared in the previous setting
2. an Arabic BERT model (I am using this to create the document representations)

from contextualized_topic_models.models.ctm import CombinedTM
from contextualized_topic_models.utils.data_preparation import TextHandler
from contextualized_topic_models.utils.data_preparation import bert_embeddings_from_list
from contextualized_topic_models.datasets.dataset import CTMDataset

handler = TextHandler(sentences=preprocessed_documents)
handler.prepare() # create vocabulary and training data

# generate BERT data
training_bert = bert_embeddings_from_list(unpreprocessed_documents, "asafaya/bert-base-arabic")

training_dataset = CTMDataset(handler.bow, training_bert, handler.idx2token)

ctm = CombinedTM(input_size=len(handler.vocab), bert_input_size=768, n_components=50)

ctm.fit(training_dataset) # run the model 

After the model has been fitted, you can get topics like this:

ctm.get_topic_lists()

I hope this helps, but feel free to ask questions if something's not clear :)

[vinid]

closing this for now :) feel free to ping me again

[nassera2014]

Thank you so much for your detailed and well-explained response, when i run the following programme

arText = []
for post in posts :
    arText.append(post['text'])

documents = arText
# documents = [line.strip() for line in open("doc.txt").readlines()]
sp = SimplePreprocessing(documents, stopwords_language = "arabic")
preprocessed_documents, unpreprocessed_documents, vocab = sp.preprocess()
print("proce : ",preprocessed_documents)
print("unproc : ",unpreprocessed_documents)
print("vocab:",vocab)

from contextualized_topic_models_master.contextualized_topic_models.models.ctm import CombinedTM
from contextualized_topic_models_master.contextualized_topic_models.utils.data_preparation import TextHandler
from contextualized_topic_models_master.contextualized_topic_models.utils.data_preparation import bert_embeddings_from_list
from contextualized_topic_models_master.contextualized_topic_models.datasets.dataset import CTMDataset

handler = TextHandler(sentences=preprocessed_documents)
handler.prepare() # create vocabulary and training data
from transformers import AutoTokenizer, AutoModel

# generate BERT data
training_bert = bert_embeddings_from_list(unpreprocessed_documents, "aubmindlab/bert-base-arabertv01")
# asafaya/bert-base-arabic
training_dataset = CTMDataset(handler.bow, training_bert, handler.idx2token)

ctm = CombinedTM(input_size=len(handler.vocab), bert_input_size=768, n_components=50)

ctm.fit(training_dataset) # run the model
print('topics : ',ctm.get_topics())

i get this error :

`/home/nassera/PycharmProjects/Facebook/venv/lib/python3.6/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
return torch._C._cuda_getDeviceCount() > 0

FileNotFoundError: [Errno 2] No such file or directory: '/home/nassera/.cache/torch/sentence_transformers/sbert.net_models_aubmindlab_bert-base-arabertv01' `

Note that i use Python 3.6
Virtual Machine : Ubuntu 18.10
and my dataset is stored into MongoDB database.

Thank you.

[silviatti]

Hi!
We tried to replicate the code in a colab notebook and used AraBERT. It seems to work. Please check and upgrade your version of the contextualized-topic-model and sentence-transformer packages. This might solve your problem :)

Silvia