-
Notifications
You must be signed in to change notification settings - Fork 2
/
README.txt
308 lines (236 loc) · 10.4 KB
/
README.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
FeatureSpaceTree
==========
Author: Adrian Pastor Lopez-Monroy, M.Sc., <pastor@ccc.inaoep.mx>
Language Technologies Lab,
Department of Computer Science,
Instituto Nacional de Astrofísica, Óptica y Electrónica
Copyright (C) 2011-2012 FeatureSpaceTree Project
For license information, see LICENSE.txt
Welcome to FeatureSpaceTree,
I created FeatureSpaceTree because I wanted an easy way to perform
text-preprocessing in text-classification tasks.
FeatureSpaceTree is a framework for scientists and students of computer science
that facilitates the textual-feature-extraction and representation of documents.
These are two common previous stages in text-classification.
Currently there are very powerful tools to perform machine learning in a very
simple way (e.g. Weka on Java, Spider on Matlab). However, feature-extraction
and representation of documents is another matter. Normally, we program the
modules directly in a scripting programming language (e.g. perl, python),
especially when the task is small. Other times we use some libraries that
provide many of the common functions (e.g. NLTK, Apache UIMA). These last tools
are the big boys of Natural Language Processing. However, the learning curve
can be scared, and normally we ended assembling the union of several components
and witring a lot of lines of code.
FeatureSpaceTree aims to be a tool for feature-extraction and representation of
documents. The main idea is just stating what features you want to extract and
some representation. This is, on the command line (future GUI) we chose
the terms (plus multiple filters used before and after the extraction) and
which representation will be used (using of multiple parameters for each
representation). and the better is, ...
Without programming a single line of code!!! ... obviously you can also use the
API inside your own code :-)
INSTALLATION
============
You will need the pip software (also needed to install NLTK) in order to install
FeatureSpaceTree.
Type into a terminal the following command:
$ sudo pip install git+https://github.com/beiceman/FeatureSpaceTree.git#egg=FeatureSpaceTree
If no errors, then congratulations you can use FeatureSpaceTree:
$ simple_exp yaml_file_with_parameters.yaml
INTRODUCTION
============
The main idea is that you will specify the following data:
1. - What attributes are needed to extract from the document corpus
(punctuation, words, n-grams at the character level, n-grams at word level,
word length, sentence length).
2. - Which filters will be applied to extract each of the attributes
(the n most frequent, n random, etc.). You could use filter on the text
before the term is extracted (e.g. eliminate capital letters). You could use
filter on tokens extracted (e.g. apply lematization).
3. - What representations will be used for these attributes, for example:
* Bag of Words (BoW)
* Latent Semantic Analysis (LSA)
* Concise Semantic Analysis (CSA)
* Bag of Terms (BoT)
* Document Ocurrence Representation (DOR)
* Term Cocurrence Representation (TCOR)
* Although the main idea is that SpaceTree works as a whole,
it is important to say that at the same time it is flexible to:
1. - Use each module independently.
2. - Extend the functionality with new modules of terms, filters
and representations.
The output of SpaceTree will be:
1.- The representation of documents in a know format (currently just weka ".arff"
and CSV format is available, however you can easily add other custom formats).
2.- A Yaml file which specify the edges of the spaces for each attribute
(ok better see the example :-) ).
3.- The vocabulary corpus for each space of attributes
4.- Some files with useful information to analyze.
VERSION
=======
.01
HOW IT WORKS?
=============
Initially SpaceTree works through the command line interface. It runs the main script and
parameters are provided through a YAML file (thanks to this, also would be easy to develop
a GUI that only treat YAML files).
$ simple_exp /somewhere/my_parameters.yaml
An example of a YAML file is:
config_base:
categories: [cat1, cat2, cat3]
experiment_base_path: /home/aplm/Experimentos/validaciones/bow/exp_validacion
experiment_name: exp_validacion
processing_option: EnumTermsProcessing.SIMPLE
corpus:
type_corpus: EnumCommonTemplate.TRAIN_TEST
train_corpus:
corpus_path: /home/aplm/nltk_data/corpora/validacion/train
filters_corpus:
- type_filter_corpus: EnumFiltersCorpus.FULL
test_corpus:
corpus_path: /home/aplm/nltk_data/corpora/validacion/test
filters_corpus:
- type_filter_corpus: EnumFiltersCorpus.FULL
# here there are parameters that can be loaded for your classification program.
# For example, the following parameters are needed for a java program based on weka.
# It means, perform 1 fold, with four classifiers into a voting scheme.
java_args:
n_folds: 10
n_classifiers: 1
classifiers_options:
- classifier: SVM
ensemble: SINGLE
# Once the corpus is specified, you will tell to SpaceTree what you want to extract.
# It is done through a tree scheme, where in the nodes you specify what attributes
# are going to be extracted and filtered. An in the leafs you will specify How
# those attributes will be represented.
# ==========================================================================
# root
# ==========================================================================
root:
terms:
# term
- type_term: EnumTermLex.REG_EXP
id_term: '1'
regexp: "[a-zA-Z'ÁÉÍÓÚáéíóúñÑüÜ]+-*[a-zA-Z'ÁÉÍÓÚáéíóúñÑüÜ]+|[a-zA-Z'ÁÉÍÓÚáéíóúñÑüÜ]+|[.]+|[/,$?:;!()&%#=+{}*~.]+|[0-9]+"
string: ''
lazy: true
mode: 0
raw_string_normalizers:
- {type_raw_string_normalizer: EnumDecoratorRawStringNormalizer.TO_LOWER}
filters_terms:
- {type_filter_terms: EnumFiltersTermsList.FIXED_TOP, fixed_top: 20000}
filters_terms:
- {type_filter_terms: EnumFiltersTermsList.FIXED_TOP, fixed_top: 20000}
childs:
# ==========================================================================
# subspace 1
# ==========================================================================
- representation: EnumRepresentation.BOW
terms:
# term
- id_term: "1"
filters_terms:
- {type_filter_terms: EnumFiltersTermsList.FIXED_TOP, fixed_top: 20000}
# space filters terms
filters_terms:
- {type_filter_terms: EnumFiltersTermsList.FIXED_TOP, fixed_top: 20000}
childs: []
# ==========================================================================
################################################################################
Another example could be:
config_base:
categories: [classname_1, classname_2, classname_n]
experiment_base_path: /somewhere/my_firts_experiment
experiment_name: my_firts_experiment
processing_option: EnumTermsProcessing.SIMPLE
test_corpus:
corpus_path: /somewhere/my_corpus/test_documents
filters_corpus:
- type_filter_corpus: EnumFiltersCorpus.IMBALANCE # How many documents to test for each class?
imbalance: [100, 100, 100]
train_corpus:
corpus_path: /somewhere/my_corpus/train_documents
filters_corpus:
- type_filter_corpus: EnumFiltersCorpus.IMBALANCE # How many documents to train for each class?
imbalance: [100, 100, 100]
# here there are parameters that can be loaded for your classification program.
# For example, the following parameters are needed for a java program based on weka.
# It means, perform 1 fold, with four classifiers into a voting scheme.
java_args:
n_folds: 1
n_classifiers: 4
classifiers_options:
- classifier: SVM
- classifier: SVM
- classifier: SVM
- classifier: SVM
ensemble: SINGLE
# Once the corpus is specified, you will tell to SpaceTree what you want to extract.
# It is done through a tree scheme, where in the nodes you specify what attributes
# are going to be extracted and filtered. An in the leafs you will specify How
# those attributes will be represented.
# ==========================================================================
# root
# ==========================================================================
root:
terms:
# term
- type_term: EnumTermLex.REG_EXP
id_term: '1'
regexp: "[a-zA-Z']+-*[a-zA-Z']+"
boolStem: false
string: ''
lazy: true
mode: 0
filters_terms:
- {type_filter_terms: EnumFiltersTermsList.FIXED_TOP, fixed_top: 200}
- type_term: EnumTermLex.TOKEN_LEN
id_term: '2'
regexp: "[a-zA-Z']+-*[a-zA-Z']+"
boolStem: false
string: ''
template: "Token{len:%s}"
lazy: true
mode: 0
filters_terms:
- {type_filter_terms: EnumFiltersTermsList.FIXED_TOP, fixed_top: 200}
# term
- type_term: EnumTermLex.BIGRAM
id_term: '3'
regexp: "[a-zA-Z']+-*[a-zA-Z']+"
boolStem: true
string: ''
lazy: true
mode: 0
filters_terms:
- {type_filter_terms: EnumFiltersTermsList.FIXED_TOP, fixed_top: 600}
filters_terms:
- {type_filter_terms: EnumFiltersTermsList.FIXED_TOP, fixed_top: 1000}
childs:
# ==========================================================================
# subspace 1
# ==========================================================================
# As this is a leaf, then you will specify you representation (BOW = Bag of Terms).
- representation: EnumRepresentation.BOW
# As this is a leaf, then you will specify which attributes wants for this representation.
terms:
# term
- id_term: "1"
filters_terms: # This is "I want the 200 most frequent tokens"
- {type_filter_terms: EnumFiltersTermsList.FIXED_TOP, fixed_top: 200}
# term
- id_term: "2"
filters_terms: # This is "I want the 200 most frequent length of tokens"
- {type_filter_terms: EnumFiltersTermsList.FIXED_TOP, fixed_top: 200}
# term
- id_term: "3"
filters_terms: # This is "I want the 200 most frequent bigrams"
- {type_filter_terms: EnumFiltersTermsList.FIXED_TOP, fixed_top: 600}
# space filters terms
filters_terms: # this is the total of terms in this subspace and in the BOW
- {type_filter_terms: EnumFiltersTermsList.FIXED_TOP, fixed_top: 1000}
childs: []
# ==========================================================================
# Here you could specify another subspace that uses some of the attributes in
# the root, but using other filters and representations.