Handwritten math expression parser
Switch branches/tags
Nothing to show
Clone or download
falvaro Model finally published
Free access to the paper until January 16 in this link http://authors.elsevier.com/a/1S6dP77nKG08n
Latest commit 84c677c Nov 30, 2015
Permalink
Failed to load latest commit information.
Config Full source code and updated README Jul 18, 2014
SampleMathExps Full source code and updated README Jul 18, 2014
rnnlib4seshat Full source code and updated README Jul 18, 2014
.gitignore Initial commit Apr 15, 2014
LICENSE Initial commit Apr 15, 2014
Makefile Full source code and updated README Jul 18, 2014
README Full source code and updated README Jul 18, 2014
README.md Model finally published Nov 30, 2015
cellcyk.cc Full source code and updated README Jul 18, 2014
cellcyk.h Full source code and updated README Jul 18, 2014
duration.cc Full source code and updated README Jul 18, 2014
duration.h Full source code and updated README Jul 18, 2014
featureson.cc Full source code and updated README Jul 18, 2014
featureson.h Full source code and updated README Jul 18, 2014
gmm.cc Full source code and updated README Jul 18, 2014
gmm.h Full source code and updated README Jul 18, 2014
gparser.cc Full source code and updated README Jul 18, 2014
gparser.h Full source code and updated README Jul 18, 2014
grammar.cc Full source code and updated README Jul 18, 2014
grammar.h Full source code and updated README Jul 18, 2014
hypothesis.cc Full source code and updated README Jul 18, 2014
hypothesis.h Full source code and updated README Jul 18, 2014
logspace.cc Full source code and updated README Jul 18, 2014
logspace.h Full source code and updated README Jul 18, 2014
meparser.cc Full source code and updated README Jul 18, 2014
meparser.h Full source code and updated README Jul 18, 2014
online.cc Full source code and updated README Jul 18, 2014
online.h Full source code and updated README Jul 18, 2014
production.cc Full source code and updated README Jul 18, 2014
production.h Full source code and updated README Jul 18, 2014
sample.cc Full source code and updated README Jul 18, 2014
sample.h Full source code and updated README Jul 18, 2014
segmentation.cc Full source code and updated README Jul 18, 2014
segmentation.h Full source code and updated README Jul 18, 2014
seshat.cc Full source code and updated README Jul 18, 2014
sparel.cc Full source code and updated README Jul 18, 2014
sparel.h Full source code and updated README Jul 18, 2014
stroke.cc Full source code and updated README Jul 18, 2014
stroke.h Full source code and updated README Jul 18, 2014
symfeatures.cc Full source code and updated README Jul 18, 2014
symfeatures.h Full source code and updated README Jul 18, 2014
symrec.cc Minor changes Oct 22, 2014
symrec.h Minor changes Oct 22, 2014
tablecyk.cc Full source code and updated README Jul 18, 2014
tablecyk.h Full source code and updated README Jul 18, 2014

README.md

SESHAT: Handwritten math expression parser

Seshat is an open-source system for recognizing handwritten mathematical expressions. Given a sample represented as a sequence of strokes, the parser is able to convert it to LaTeX or other formats like InkML or MathML. You can see an example of application of this parser in

http://cat.prhlt.upv.es/mer/

where seshat is the underlying engine.

This parser has been developed by Francisco Álvaro as part of his PhD thesis while he was a member of the [PRHLT research center] 1 at [Universitat Politècnica de València] 2.

Seshat represents a state-of-the-art system that has participated in several [international competitions] 3, and it was awarded the best system trained on the competition dataset in:

  • Mouchère H., Viard-Gaudin C., Zanibbi R., Garain U. ICFHR 2014 Competition on Recognition of On-line Handwritten Mathematical Expressions (CROHME 2014). International Conference on Frontiers in Handwriting Recognition (ICFHR), Crete Island, Greece (2014)

The math expression recognition model that seshat implements is described in:

  • Francisco Álvaro, Joan-Andreu Sánchez and José-Miguel Benedí. An Integrated Grammar-based Approach for Mathematical Expression Recognition. Pattern Recognition, pp. 135-147, 2016

and it is the main part of my PhD thesis.

License

Seshat is released under the [GNU General Public License version 3.0 (GPLv3)] 5

Distribution details

Seshat is written in C++ and should work on any platform, although it has only been tested in Linux.

This software integrates the open-source [RNNLIB library] 4 for symbol classification. The code of RNNLIB has been slightly modified and directly integrated in seshat, thus, it is not necessary to download it. However, it requires the [Boost C++ Libraries] 6 (headers only).

Finally, the parser accepts input files in two formats: InkML and SCGINK. There is a example of each format in folder "SampleMathExps". Seshat uses the [Xerces-c library] 7 for parsing InkML in C++.

Installation

Seshat is written in C++ and it only requires Makefile and g++ to compile it. Once the required tools and libraries are available, you can proceed with the installation of seshat as follows:

  1. Obtain the package using git:

    $ git clone https://github.com/falvaro/seshat.git
    

    Or [download it as a zip file] 8

  2. Go to the directory containing the source code.

  3. If the include files of boost libraries are not in the path, add it to the FLAGS variable in the file Makefile ("-I/path/to/boost/").

  4. Compile seshat

    $ make

As a result, you will have the executable file "seshat" ready to recognize handwritten math expressions.

Example of usage

Run seshat without arguments and it will display the command-line interface:

$ Usage: ./seshat -c config -i input [-o output] [-r render.pgm]

  -c config: set the configuration file
  -i input:  set the input math expression file
  -o output: save recognized expression to 'output' file (InkML format)
  -r render: save in 'render' the image representing the input expression (PGM format)
  -d graph:  save in 'graph' the description of the recognized tree (DOT format)

There are two example math expressions in folder "SampleMathExps". The following command will recognize the expression (x+y)^2 encoded in "exp.scgink"

$ ./seshat -c Config/CONFIG -i SampleMathExps/exp.scgink -o out.inkml -r render.pgm -d out.dot

This command outputs several information through the standard output, where the last line will provide the LaTeX string of the recognized math expression. Furthermore:

  • An image representation of the input strokes will be rendered in "render.pgm".

  • The InkML file of the recognized math expression will be saved in "out.inkml".

  • The derivation tree of the expression provided as a graph in DOT format will be saved in "out.dot". The representation of the graph in, for example, postscript format can be obtained as follows

     	  $ dot -o out.ps out.dot -Tps
    

It should be noted that only options "-c" and "-i" are mandatory.

Citations

If you use seshat for your research, please cite the following reference:

@article{AlvaroPR16,
title = "An integrated grammar-based approach for mathematical expression recognition",
author = "Francisco \'Alvaro and Joan-Andreu S\'anchez and Jos\'e-Miguel Bened\'{\i}",
journal = "Pattern Recognition",
volume = "51",
pages = "135 - 147",
year = "2016",
issn = "0031-3203"
}

Why seshat?

Seshat was the [Goddess of writing] 9 according to Egyptian mythology, so I liked this name for a handwritten math expression parser. I found out about seshat because my colleague of the PRHLT [Daniel Ortiz-Martínez] 10 developed [Thot] 11, a great open-source toolkit for statistical machine translation; and Thot is the [God of Knowledge] 12 according to Egyptian mythology.