Language is “...a systematic means of communicating ideas or feelings by the use of conventionalized signs, sounds, gestures, or marks having understood meanings”. In natural language processing, we try to come up with algorithms that can parse information, facts and sentiments quickly, efficiently and accurately from written text or speech. The fundamental problem in solving any of these tasks involve understanding the language itself, specifically its semantics and grammar. Language modeling addresses this key issue - it is a statistical measure of whether a sequence of words semantically makes sense in a particular language. In this report we will take a look at how we can model the English language using a deep learning architecture called Recurrence Neural Network (RNN) with Long-Short Term Memory (LSTM) units. Two modeling approaches are explored - one at a word level and another at a character level. The report compares and contrasts the two approaches through exhaustive experiments and identify the tradeoff and limitations of both the approaches.
- Use setup.sh to install required packages.
- Change hyper parameters in config.py
- For training word based model
- For character based model
- Models are saved in character_models OR word_models folder