added support word2vec training with additional data #18636
Conversation
|
Hi there - I don't see the value here of adding a few words in a String array to the training. You're effectively adding a second (non-distributed, therefore limited in size) corpus to the training. Word2Vec is more aimed at training on a larger corpus of text. If you want more accuracy train on a larger training set. Could you close this PR please? |
|
At the moment, it is not possible to improve a models accuracy by incorporating additional data. I think this should be supported since it can increase a classifiers performance significantly. With this implementation, I was able to train unsupervised on a Wikipedia Dump, which is pretty large. However, distributing the set is a good point. |
|
I'm sorry but I still don't understand the intention here. You can already train on a Wikipedia dump (or any other dataset) by passing that dataset as the input DataFrame to Word2Vec. If you want to "incorporate additional data" why not just |
|
The problem emerges in cases where you built a whole pipeline. You have a set of documents you want to classify. These documents have some additional features and they are preprocessed in the pipeline. When coming to Word2Vec, you want to vectorize your documents. However, you see bad performance of your word vectors and you want to tune them by adding additional documents. You don't want these documents to be part of the whole pipeline, because they are unable to pass the previous preprocessing steps. That was my intention to add this. Probably, it is a very rare usecase. I don't know. |
|
Can one of the admins verify this patch? |
What changes were proposed in this pull request?
Word2Vec is trained unsupervised. The more data it is trained on, the more "accurate" are the word vectors. Hence, Word2Vec should support to be fit on additional data.
How was this patch tested?
Additional unit test.
Please review http://spark.apache.org/contributing.html before opening a pull request.