Skip to content
Unine light stemmer for French, German, Italian, Spanish, Portuguese, Finnish, Swedish
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
R
data
docs
man
src
tests
tools
.Rbuildignore
.gitattributes
.gitignore
.travis.yml
CRAN-RELEASE
DESCRIPTION
LICENSE
LICENSE.md
NAMESPACE
NEWS.md
README.md
appveyor.yml
codecov.yml
cran-comments.md
unine.Rproj

README.md

UNINE

Travis build status Build status Coverage status CRAN status CRAN_Download lifecycle License: MIT Follow

Implementation of "light" stemmers for French, German, Italian, Spanish, Portuguese, Finnish, Swedish.
They are based on the same work as the "light" stemmers found in SolR or ElasticSearch.
A "light" stemmer consists in removing inflections only for noun and adjectives.
Indexing verbs for these languages is not of primary importance compared to nouns and adjectives.

The procedures used in this stemmer are described below:

  • the stemming procedure for French is described in (Savoy, 1999).
  • in Italian, the main inflectional rule is to modify the final character (e.g., «-o», «-a» or «-e») into another (e.g., «-i», «-e»). As a second rule, Italian morphology may also alter the final two letters (e.g., «-io» in «-o», «-co» in «-chi», «-ga» in «-ghe»).
  • in German, a few rules may be applied to obtain the plural form of words (e.g., "Frau" into "Frauen" (woman), "Bild" into "Bilder" (picture), "Sohn" into "Söhne" (son), "Apfel" into "Äpfel" (apple)), but the suggested algorithms do not account for person and tense variations, or for the morphological variations used by verbs.

Online tests are available on this website.

Installation

You can install the released version of unine from CRAN with:

install.packages("unine")

... or the last version from Github

devtools::install_github("pommedeterresautee/unine")

Example

Below some examples for French and a comparaison with Porter French stemmer.

french_stemmer(words = c("complète", "caissière"))
# [1] "complet"  "caisier"
# Not that below double letters are deduplicated: caissière -> caisier

french_stemmer(words = c("tester", "testament", "chevaux", "aromatique", "personnel", "folle"))
# [1] "test"      "testament" "cheval"    "aromat"    "personel" "fou" 
# Not that below double letters are deduplicated: personnel -> personel

# look at how "testament" and "tester" have been stemmed above. 
# Now with Porter stemmer :
SnowballC::wordStem(c("testament", "tester"), language = "french")
# [1] "test" "test"

References

Please cite [1] if using this R package.

[1] J. Savoy, A stemming procedure and stopword list for general French corpora

@article{savoy1999stemming,
  title={A stemming procedure and stopword list for general French corpora},
  author={Savoy, Jacques},
  journal={Journal of the American Society for Information Science 50(10), 944-952.},
  year={2009}
}
You can’t perform that action at this time.