Release 3.16.0

Noteworthy changes in 3.16.0

• Bugfixes:

  • During conversion between optimized-lookup and other formats, the alphabet
    was not being preserved, leading to incorrect behaviour in some cases.
    This has been fixed.

  • Fix segfaults in hfst-summarize, hfst-disjunct and foma back-end.

  • Force ICU, fixes #501.

  • Use unique_ptr for input/ouput stream memory allocation (#547).

• Improvements:

  • hfst-pmatch archives now no longer include or parse redundant alphabets,
    which in some cases can be a meaningful size and speed saving.

  • Multiple memory leaks fixed.

  • Drop support for TR1 standard assuming that C++14 is minimal requirement.

  • Drop legacy wrappers for foma and lexc (#532).

  • Use upstream Foma and by default upstream OpenFST.

  • Use C++17, or C++14 with bundled OpenFST.

  • Update external links for sfst, foma and flex.

  • Add lex/yacc sources to dist (see issue #524).

• Features:

  • A new variable in the hfst-pmatch language, "unicode-character-classes",
    which causes the use of built-in character sets to be interpreted at
    runtime using a unicode library.

  • Verbose output from hfst-pmatch2fst has been much improved, providing eg.
    a call graph of the process. It also warns about use of pmatch features
    that is likely to be incorrect and lead to errors.

The file hfst-3.16.0.tar.gz below contains the source code.
For installation instructions and alternative installation methods, see our download page.

Release 3.15.5

Noteworthy changes in 3.15.5

• Fixes to pmatch:

  • Modify tracking of entry points to allow for RTNs in Captures()

  • Fix off-by-one bug in checking captures

• Fixes to other issues: #508, #488, #486, #452, #491, #494

• Fix memory leaks

The file hfst-3.15.5.tar.gz below contains the source code.
For installation instructions and alternative installation methods, see our download page.

Release 3.15.4

Intended license was LGPLv3+ so formalize that, and explicitly mentio…

…n that the SFST-derived files are used under LGPLv3+ agreement (closes #494)

Release 3.15.3

v3.15.3

accoring -> according

Release 3.15.2

Noteworthy changes in 3.15.2

• Fix off-by-one bug in checking captures in pmatch

• Fix issues #443, #429 and #409

The file hfst-3.15.2.tar.gz below contains the source code.
For installation instructions and alternative installation methods, see our download page.

Release 3.15.0

Noteworthy changes in 3.15.0

• Numerous improvements in pmatch processing:

  • hfst-pmatch2fst now implements eg @include"filename.txt" as a preprocessing stage

  • hfst-pmatch2fst now implements Explode() and Implode() in full generality

  • hfst-pmatch now allows printing weights (in location mode, with --locate)

  • hfst.PmatchContainer.locate offered via python bindings

  • bugfixes and improvements hfst.PmatchContainer.tokenize

• Add tool hfst-eliminate-flags.

• Allow both escaped and unescaped zeros in multicharacters in all lexc lexicon entries (fixes issue #274).

• Also reweight state that is both initial and final in hfst-reweight (fixes issue #368).

• Do not let distributed files depend on configuration. Include man files in dist.

• Print warnings and error messages also in earlier stages of twol compilation.

• Add option --execute-and-quit (-E) to hfst-xfst.

The file hfst-3.15.0.tar.gz above contains the source code.
For installation instructions and alternative installation methods, see our download page.

Release 3.14.0

Noteworthy changes in 3.14.0

• Numerous improvements to pmatching and tokenization:

  • pmatch now supports the default symbol

  • pmatch now supports reading word embeddings in the binary format

  • improvements to pmatch runtime context handling: a bug affecting expression-initial contexts is fixed, and pmatch now supports Ins() arcs inside RC() and NRC() contexts

  • in pmatch, bugs affecting multiple Ins() arcs, in particular nested ones, are fixed

• Implement variable 'retokenize' in hfst-xfst

The file hfst-3.14.0.tar.gz contains the source code.
For installation instructions and alternative installation methods, see our download page.

Release 3.13.0

Noteworthy changes in 3.13.0

• Numerous improvements to pmatching and tokenization:

  • "[].with(X = Y)" feature in pmatch. This provides support in the pmatch2fst compiler to define "global flags".

  • Add a variable "xerox-composition", default to "on".

  • Consider list symbols ("@l..." and "@x...") to be special.

  • Fix runtime handling of contexts and compilation of negative contexts.

  • Make Like() and Unlike() much faster by not sorting the whole vocabulary and only calculating each comparison key once.

  • Keep track of weights along context checking paths and unify weight handling.

  • In blankline-separated mode, keep blanklines in output too.

  • Round weight to zero decimals, non-sci.

  • Make hfst-tokenise usable as a lib; include simple string-to-string function.

  • Use libreadline in hfst-pmatch when available.

  • Cleanup remainder of the pmatch test suite, all the tests now pass.

• Python interface:

  • Support reading several twolc files.

  • Add functions 'compose' and 'cross_product' that take a list of transducers.

  • Allow empty string as input for hfst.fst and hfst.fsa and interpret it as epsilon.

  • Perform fsmbook tests also via python API.

  • Add option --local-hfst to setup.py.

  • Include pre-swig-generated wrappers to pypi source distribution.

• Compilation:

  • Use by default c++11 unordered_map and unordered_set, unless otherwise specified.

  • Add an option --without-c++11 (defaults to 'no') to compile hfst without c++11 support.

  • Require libc++ and osx version >= 10.7 with clang.

• New functions and options:

  • Add function HfstBasicTransducer::remove_final_weight.

  • Add function HfstTransducer::negate() for automata.

  • Add option --restricted-mode (-R) to hfst-xfst.

  • Flag diacritics: support getting a list of operations involved with a particular feature.

  • Allow creating HfstInputStream's from std::istream's.

• Fix issues #341 and #353, make workarounds for issue #358.

The file hfst-3.13.0.tar.gz below contains the source code.
For installation instructions and alternative installation methods, see our download page.

Release 3.12.2

Noteworthy changes in 3.12.2

• Changes to configure:

  • Disable lexc and foma wrappers as well as hfst-train-tagger tool unless explicitly requested

  • Enable hfst-calculate and hfst-xfst by default

  • Add experimental options --with-openfst-log=lean and --with-sfst=lean which support a limited nuber of operations for these types (reading, writing, converting between t
    ypes and deleting)

  • Require at least automake version 1.12 unless compiling from pre-yacc-generated sources

• Implement hfst-twolc as a single program instead of a script. Get rid of hfst-twolc-system and hfst-twolc-loc tools.

• Improve pmatch compilation and error handling mechanisms

• Improve hfst-tokenize tool

• Add transliterate output mode (--transliterate) to hfst-proc

• Changes to python interface:

  • Support twolc and sfst compilation

  • Improve HfstBasicTransducer iteration mechanism

  • Add experimental support for python version 2

  • Support apply up and apply down commands in function start_xfst

The file hfst-3.12.2.tar.gz below contains the source code.
For installation instructions and alternative installation methods, see our download page.

Release 3.12.1

Noteworthy changes in 3.12.1

• Fix flag elimination bug (reported in issue #342)
• Do not allow unescaped dots in regular expressions
• Improvements to pmatch and tokenization tools:
  • Search for included files under scriptdir, not working dir
  • Add experimental two-vector model for word sense
  • Handle Apertium-style superblanks in --giella-cg format
• Rename the PyPI package to 'hfst' (available at https://pypi.python.org/pypi/hfst)
• Update foma back-end

The file hfst-3.12.1.tar.gz below contains the entire source code.
The file libhfst-3.12.1.tar.gz is a minimal source code package which contains only hfst library, openfst back-end, and python bindings. It is licensed as LGPL.
The files hfst-3.12.1_py3.3-3.4_win64 contain 64-bit python bindings for python versions 3.3 and 3.4 for windows.
For installation instructions and alternative installation methods, see our download page.