T-scan: an analysis tool for dutch texts to assess the complexity of the text, based on original work by Rogier Kraf
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
config modified config to work with recent frog Dec 21, 2017
data updated download script. Something goes wrong on MacOSX Apr 10, 2018
docs Updated the manual Mar 31, 2017
include Added features for normalization of Wopr scores Jun 6, 2018
m4 modernized configure.ac Apr 10, 2018
src Added features for normalization of Wopr scores Jun 6, 2018
tests Merge branch 'master' of github.com:proycon/tscan Apr 10, 2018
view protocol-relative URLs Sep 2, 2015
webservice Version bump after release Jun 6, 2018
.gitignore added README to .gitignore cause it the symlink bootstrap creates (to… Apr 6, 2018
.travis.yml updated .travis.yml. Include cppcheck and scan-buid. Also attempt a M… Apr 10, 2018
AUTHORS added Martijn van der Klis to AUTHORS, downgraded myself in AUTHORS l… May 27, 2015
COPYING License change from GPLv3 to AGPLv3 Aug 27, 2015
Makefile.am add the download file to the distribution. Mar 9, 2016
NEWS Added a (non fuctional, but working) distribution Sep 10, 2012
README added README Apr 4, 2018
README.md Full path for adjs_semtype and nouns_semptype needed, fixes #7 Apr 4, 2018
bootstrap.sh small adaptation to make it work with a more recent ticcutils Apr 4, 2018
codemeta.json affiliation fix in metadata Jun 13, 2018
configure.ac Version bump after release Jun 6, 2018
downloaddata.sh oepsie, a typo Apr 10, 2018
surprise.sh revert Dec 18, 2012
tscan.cfg.example Removing the (incomplete) LSA calculation from the project Apr 10, 2018

README.md

Build Status Language Machines Badge

T-Scan

tscan 0.9 (c) TiCC/ 1998 - 2018

Tilburg centre for Cognition and Communication, Tilburg University.
UiL-OTS, Utrecht University
Language Machines, Centre for Language Studies, Nijmegen

T-Scan is distributed under the GNU Affero Public Licence (see the file COPYING).

T-Scan is an analysis tool for dutch texts to assess the complexity of the text, and is based on original work by Rogier Kraf (Utrecht University) (see: Kraf et al., 2009). The code has been reimplemented and extended by Ko van der Sloot (Tilburg University), and is currently maintained and continued by Martijn van der Klis (Utrecht University).

Web application / Webservice

This repository contains the T-Scan source code, allowing you to run it yourself on your own system. In addition, T-Scan is available as a web application and webservice through https://webservices-lst.science.ru.nl , register for a (free) account there first, and then access T-Scan through https://webservices-lst.science.ru.nl/tscan/ .

Documentation

Extensive documentation (in Dutch) can be found in docs/tscanhandleiding.pdf.

Installation

T-Scan heavily depends upon other sofware, such as Frog, Wopr and Alpino.

Installation is not trivial, to be able to succesfully build tscan from the tarball, you need the following packages:

  • autotools
  • autoconf-archive
  • ticcutils
  • libfolia
  • Alpino
  • frog
  • wopr
  • CLAM

To facilitate installation, T-Scan is included as an extra option in LaMachine

We strongly recommend to use LaMachine to install tscan. In addition, T-Scan also uses Alpino, which is also included in LaMachine.

To install T-scan in an existing LaMachine environment you may need to adapt your installation manifest, as it is not included by default:

(lamachine)$ lamachine-update --edit

If you do not want to use LaMachine, first make sure you have all necessary dependencies and then compile/install as follows:

$ bash bootstrap.sh
$ ./configure
$ make
$ sudo make install
$ cd webservice
$ python3 setup.py install

Usage

If you use LaMachine as recommended, always activate the virtual environment first.

$ source lamachine-activate

Before you can use T-Scan you need to start the background servers (you may need to edit the scripts to set ports and paths):

$ cd tscan/webservices
$ ./startalpino.sh
$ ./startfrog.sh
$ ./startwopr20.sh    (will start Wopr to calculate forwards probabilities)
$ ./startwopr02.sh    (will start Wopr to calculate backwards probabilities)

Then either run tscan from the command-line, which will produce a FoLiA XML file,

$ cd tscan
$ cp tscan.cfg.example tscan.cfg
(edit tscan.cfg if necessary)
$ tscan --config=tscan.cfg input.txt

... or use the webapplication/webservice:

$ clamservice ticclservice.tscan   #this starts the CLAM service for TSCAN

And then navigate to the host and port specified.

Data

Word prevalence values (in data/prevalence_nl.data and data/prevalence_be.data) courtesy of Keuleers et al., Center for Reading Research, Ghent University.

Certain parts of T-Scan use data from Referentiebestand Nederlands, which we can not distribute due to restrictive licensing issues, so this functionality will not be available.

Certain other data is too large for GitHub, but will be downloaded for you automatically by the ./downloaddata.sh script.

References

  • Kraf, R. & Pander Maat, H. (2009). Leesbaarheidsonderzoek: oude problemen en nieuwe kansen. Tijdschrift voor Taalbeheersing 31(2), 97-123.
  • Pander Maat, H. & Kraf, R. & van den Bosch, A. & Dekker, N. & van Gompel, M. & Kleijn, S. & Sanders, T. & van der Sloot, K. (2014). T-Scan: a new tool for analyzing Dutch text. Computational Linguistics in the Netherlands Journal 4, 53-74.
  • Keuleers, E. & Stevens, M. & Mandera, P. & Brysbaert, M. (2015). Word knowledge in the crowd: Measuring vocabulary size and word prevalence in a massive online experiment. The Quarterly Journal of Experimental Psychology 68(8), 1665-1692.