Analysis and improvements of the language tagging of LOD Laundromat literals
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
datatype_catalogs
langdetect-03-03-2014
make_paper_tables
old_output
.gitignore
README.md
algorithm_multi_run.sh
algorithm_run_curl.sh
cld_algorithm.js
cld_algorithm_run_fs.sh
ld_algorithm.js
ld_algorithm_run_fs.sh
md5s_with_ext.data
no_library_multi_run.sh
nolibrary.js
nolibrary_run_fs.sh
tika_algorithm.js
tika_algorithm_run_fs.sh

README.md

Analysis and improvements of the language tagging of LOD Laundromat literals, supporting the paper "Literally better: Analyzing and Improving the Quality of Literals".

Directory structure

The algorithm_multi_run.sh is the main script to be executed. It fires all three libraries on the entire LOD Laundromat collection.

For each of the libraries (cld, ld, tika), there are two files that execute them and tag a document: the scripts [LIBRARY]_algorithm_fs.sh and the nodejs scripts [LIBRARY]_algorithm.js.

Analogue to this, there are scripts that execute no library, but instead count the tagged and untagged strings in the original data. These are no_library_multi_run.sh, nolibrary_run_fs.sh, and nolibrary.js.

output/ contains the output summaries by each of the libraries on each document in the LOD Laundromat. It also contains the nolib/ directory that only counts the amount of tagged and untagged strings per document.

make_paper_tables/ is a directory that contains the scripts to adapt the processing output to the needs of the paper, i.e. this produces the tables as found in the paper based on the original JSON data in the folder output/.

datatype_catalogs/ contains the frequencies we obtain per datatype.

Running the code

is as simple as running the command:

sh algorithm_multi_run.sh

Note: You need to have Frank and the langdetect installed before that in the root directory of this project.

Contact

Filip Ilievski (f.ilievski@vu.nl)