-
Notifications
You must be signed in to change notification settings - Fork 3
/
RcppCWB_package.R
141 lines (140 loc) · 6.69 KB
/
RcppCWB_package.R
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
#' Rcpp Bindings for the Corpus Workbench (CWB).
#'
#' @description
#' The \code{RcppCWB} package is a wrapper library to expose core functions of
#' the \code{Open Corpus Workbench} (CWB). This includes the low-level
#' functionality of the \code{Corpus Library} (CL) as well as capacities to use
#' the query syntax of the \code{Corpus Query Processor} (CQP).
#'
#' @section The Idea Behind RcppCWB:
#'
#' The \code{Open Corpus Workbench} (CWB) is an indexing and querying engine
#' popular in corpus-assisted research. Its core aim is to support working
#' efficiently with large, structurally and linguistically annotated corpora.
#' First of all, the CWB includes tools to index and compress corpora. Second,
#' the \code{Corpus Library} (CL) offers low-level functionality to retrieve
#' information from CWB indexed corpora. Third, the \code{Corpus Query
#' Processor} (CQP) offers a syntax that allows to perform anything from
#' simple to complex queries, using different annotation layers of corpora.
#'
#' The CWB is a classical tool which has inspired a set of developments. A
#' persisting advantage of the CWB is its mature, open source code base that
#' is actively maintained by a community of developers. It is used as a robust
#' and efficient backend for widely used tools such as TXM
#' (\url{http://textometrie.ens-lyon.fr}) or CQPweb
#' (\url{http://cwb.sourceforge.net/cqpweb.php}). Its uncompromising C
#' implementation guarantees speed and makes it well suited to be integrated
#' with R at the same time.
#'
#' The package \code{RcppCWB} is a follow-up on the \code{rcqp} package that
#' has pioneered to expose CWB functionality from within R. Indeed, the
#' \code{rcqp} package, published at CRAN in 2015, offers robust access to CWB
#' functionality. However, the "pure C" implementation of the \code{rcqp}
#' package creates difficulties to make the package portable to Windows. The
#' primary purpose of the \code{RcppCWB} package is to reimplement a wrapper
#' library for the CWB using a design that makes it easier to achieve
#' cross-platform portability.
#'
#' Even though \code{RcppCWB} functions may be used directly, the package is
#' designed to serve as an interface to CWB indexed corpora in packages with
#' higher-level functionality. In this regard, \code{RcppCWB} is the backend
#' of the \code{polmineR} package. It is deliberately open to be used in other
#' contexts. The package may stimulate using linguistically annotated, indexed
#' and compressed corpora on all platforms. The paradigm of working with text
#' as linguistic data may benefit from \code{RcppCWB}.
#' @section Implementation:
#'
#' When building the package, the first step is to compile the relevant parts
#' of the CWB on Linux and macOS machines. On Windows, cross-compiled binaries
#' are downloaded from a GitHub repository of the PolMine Project
#' (\url{https://github.com/PolMine/libcl}). Second, \code{Rcpp} wrappers are
#' compiled and make the relevant functions of the Corpus Library and CQP
#' accessible. In addition to genuine CWB functions, \code{RcppCWB} offers a
#' set of higher level functions implemented using \code{Rcpp} for common
#' performance critical tasks.
#'
#'
#' @section Getting Started with RcppCWB:
#'
#' To understand the data storage model of the CWB, in particular the notions
#' of positional and structural attributes (s- and p-attributes), the vignette
#' of the \code{rcqp} package is a very good starting point (see references).
#'
#' The CWB 'Corpus Encoding Tutorial' explains how to create your own corpus,
#' the 'CQP Query Language Tutorial' introduces the syntax of CQP (see
#' references).
#'
#' The \code{RcppCWB} package includes a sample corpus (REUTERS, the data also
#' included in the \code{tm} package). The examples in the documentation
#' of the functions may be a good starting point to understand how to use
#' \code{RcppCWB}.
#'
#' @section Digging Deeper:
#'
#' The original paper of Christ (1994) explains the design choices of the CWB.
#' The indexing and compression techniques of the CWB (Huffman coding) are
#' explained in Witten et al. (1999).
#'
#' @section Acknowledgements:
#'
#' The work of the all developers of the CWB is gratefully acknowledged. There
#' is a particular intellectual debt to Bernard Desgraupes and Sylvain
#' Loiseau, and the \code{rcqp} package they developed as the original R
#' wrapper to expose the functionality of the CWB.
#'
#' @references
#' Christ, O. 1994. "A modular and flexible architecture for an integrated
#' corpus query system", in: Proceedings of COMPLEX '94, pp. 23-32. Budapest.
#' Available online at \url{http://cwb.sourceforge.net/files/Christ1994.pdf}
#'
#' Desgraupes, B.; Loiseau, S. 2012. Introduction to the rcqp package.
#' Vignette of the rcqp package. Available at the CRAN archive at
#' \url{https://cran.r-project.org/src/contrib/Archive/rcqp/}
#'
#' Evert, S. 2005. The CQP Query Language Tutorial. Available online at
#' \url{http://cwb.sourceforge.net/files/CWB_Encoding_Tutorial.pdf}
#'
#' Evert, S. 2005. The IMS Open Corpus Workbench (CWB). Corpus Encoding
#' Tutorial. Available online at
#' \url{http://cwb.sourceforge.net/files/CWB_Encoding_Tutorial.pdf}
#'
#' Open Corpus Workbench (\url{http://cwb.sourceforge.net})
#'
#' Witten, I.H.; Moffat, A.; Bell, T.C. (1999). Managing Gigabytes. Morgan
#' Kaufmann Publishing, San Francisco, 2nd edition.
#'
#'
#' @keywords package
#' @docType package
#' @rdname RcppCWB-packge
#' @aliases RcppCWB RcppCWB-package
#' @name RcppCWB-package
#' @useDynLib RcppCWB, .registration = TRUE
#' @importFrom Rcpp evalCpp
#' @exportPattern "^[[:alpha:]]+"
#' @author Andreas Blaette (andreas.blaette@@uni-due.de)
#' @examples
#' # functions of the corpus library (starting with cl) expose the low-level
#' # access to the CWB corpus library (CL)
#'
#' registry <- if (!check_pkg_registry_files()) use_tmp_registry() else get_pkg_registry()
#' ids <- cl_cpos2id("REUTERS", cpos = 1:20, p_attribute = "word", registry = registry)
#' tokens <- cl_id2str("REUTERS", id = ids, p_attribute = "word", registry = registry)
#' print(paste(tokens, collapse = " "))
#'
#' # To use the corpus query processor (CQP) and its syntax, it is necessary first
#' # to initialize CQP (example: get concordances of 'oil')
#'
#' cqp_initialize(registry)
#' cqp_query("REUTERS", query = '[]{5} "oil" []{5}')
#' cpos_matrix <- cqp_dump_subcorpus("REUTERS")
#' concordances_oil <- apply(
#' cpos_matrix, 1,
#' function(row){
#' ids <- cl_cpos2id("REUTERS", p_attribute = "word", cpos = row[1]:row[2])
#' tokens <- cl_id2str("REUTERS", p_attribute = "word", id = ids)
#' paste(tokens, collapse = " ")
#' }
#' )
#'
NULL