remove-stopword
is a node module that allows you to strip stopwords from an
input text. In natural language processing, "Stopwords" are words
that are so frequent that they can safely be removed from a text
without altering its
meaning..
This library is specifically designed for WorldBrain's usecase of stripping as many words from every webpage as possible to make search-indexing faster in regards to several thousand documents of varying information.
Credits:
This module was essentially coppied directly from @fergiemcdowall's stopword library. The only differences is that more language support was added from this stopwords json lib Also there are minor tweaks to several languages specifically for worldbrains use-case. Unless otherwise specified all the stopwords came from stopwords json lib
By default, stopword
will strip an array of "meaningless" English words
sw = require('stopword')
const oldString = 'a really Interesting string with some words'.split(' ')
const newString = sw.removeStopwords(oldString)
// newString is now [ 'really', 'Interesting', 'string', 'words' ]
You can also specify a language other than English, as a string:
sw = require('stopword')
const oldString = 'Trädgårdsägare är beredda att pröva vad som helst för att bli av med de hatade mördarsniglarna åäö'.split(' ')
// sw.sv contains swedish stopwords
const newString = sw.removeStopwords(oldString, 'sv')
// newString is now [ 'Trädgårdsägare', 'beredda', 'pröva', 'helst', 'hatade', 'mördarsniglarna', 'åäö' ]
You can also specify to remove stopwords from all languages by specifying 'all'
:
sw = require('stopword')
const oldString = 'Trädgårdsägare är beredda att a really Interesting string with some words ciao'.split(' ')
// 'all' iterates over every stopword list in the lib
const newString = sw.removeStopwords(oldString, 'all')
// newString is now [ 'Trädgårdsägare', 'beredda', 'really', 'Interesting', 'string', 'words' ]
And last, but not least, it is possible to use your own, custom list of stopwords:
sw = require('stopword')
const oldString = 'you can even roll your own custom stopword list'.split(' ')
// Just add your own list/array of stopwords
const newString = sw.removeStopwords(oldString, [ 'even', 'a', 'custom', 'stopword', 'list', 'is', 'possible']
// newString is now [ 'you', 'can', 'roll', 'your', 'own']
Arrays of stopwords for the following languages are supplied:
af
- Afrikaansar
- Modern Standard Arabichy
- Armenianeu
- Basquebn
- Bengalibr
- Brazilian Portuguesebg
- Bulgarianca
- Catalanzh
- Chinesehr
- Croationhr
- Czechda
- Danishnl
- Dutchen
- Englisheo
- Esperantoet
- Estonianfa
- Farsifi
- Finnishfr
- Frenchgl
- Galiciande
- Germanel
- Greekha
- Hausahe
- Hebrewhi
- Hindihu
- Hungarianid
- Indonesianga
- Irishit
- Italianja
- Japaneseko
- Koreanla
- Latinlv
- Latvianmr
- Marathino
- Norwegianfa
- Persianpl
- Polishpt
- Portuguesero
- Romanianru
- Russiansk
- Slovaksl
- Slovenianso
- Somaliast
- Southern Sothoes
- Spanishsw
- Swahilisv
- Swedishth
- Thaiyo
- Yorubazu
- Zulu
sw = require('stopword')
norwegianStopwords = sw.no
// norwegianStopwords now contains an Array of norwgian stopwords
ja
Japanese and zh
Chinese Simplified have no space between words. For these languages you need to split the text into words before feeding it to the stopword
module. You can check out TinySegmenter for Japanese and chinese-tokenizer for Chinese.
Returns an Array that represents the text with the specified stopwords removed.
text
An array of wordsstopwords
An array of stopwords
sw = require('stopword')
var text = sw.removeStopwords(text[, stopwords])
// text is now an array of given words minus specified stopwords