Python package for symbol/word and their bigrams frequency analysis with excel output.
What values can be counted: quantity, quantity in the first position, quantity in the last position, average position.
For which data values can be counted: symbols, symbol bigrams, words, word bigrams.
Additional possible data: ye-yo words table for Russian language (in excel output it can be cross-referenced with words quantity).
pip install frequency-analysis
;- Download data set of your choice;
- Call
Analysis
class with context manager (take a look at the optional arguments); - Parse your data set to word list (one sentence in list for properly word position counting) and send it to one of three methods of
Analysis
; - Call
Result
class with context manager (with optionalname
argument); - Call one or several of
Result
methods to create excel sheet(s) with appropriate data.
All arguments are optional
- name – the name of the folder in which the analysis will be saved
defaultfrequency_analysis
- mode – analysis operation mode – [n]ew, [a]ppend (to existing), [c]ontinue (to send the previous unfinished set to existing analysis)
defaultn
- word_pattern – regex pattern for matching inwords symbols
default'[a-zA-Zа-яА-ЯёЁ]+(?:(?:-?[a-zA-Zа-яА-ЯёЁ]+)+|'?[a-zA-Zа-яА-ЯёЁ]+)|[a-zA-Zа-яА-ЯёЁ]'
- allowed_symbols – string of symbols or list with symbol unicode decimal values, which will be counted to analysis
default[*range(32, 127), 1025, *range(1040, 1104), 1105]
(base punctuation, base Latin, Russian Cyrillic) - yo – int for additional Russian word processing – compare words with word list to detect number of ye/yo misspelling. 0 – disabled; 1 – enabled; 2 with 'a' mode – update yo list with new data.
To use this mode you should place two word files near the running script (yo.txt
for words with mandatoryyo
andye-yo.txt
for possiblyyo
writing). You can use your own or take it here.
default0
Method for counting symbol and symbol_bigram frequency.
Counted values: quantity, quantity in the first position, quantity in the last position, average position in word.
Average position counted only with argument pos
as True
(default False
).
Position for symbols, which matched with word_pattern
counted as for "clear" word, for other – as for "raw".
Example: in single word "–Yes!" with default word_pattern
positions will be counted as (– 1), (Y 1), (e 2), (s 3), (! 5).
Bigrams counting can be disabled with argument bigram
as False
(default True
).
Method for counting word and word_bigrams frequency.
Counted values: quantity, quantity in the first position, quantity in the last position, average position in sentence.
Average position counted only with argument pos
as True
(default False
).
Bigrams counting can be disabled with argument bigram
as False
(default True
).
Combined call of previous two methods.
The only argument is optional
- name – the name of the folder in which the analysis was saved
defaultfrequency_analysis
First 6 methods can be called all it once with treat() method
Many methods accept arguments limit
, chart_limit
, min_quantity
and ignore_case
.
- limit (default
0
) it is a max number of elements, which will be added to the sheet. Zero – unlimited; - chart_limit (default
20
) – a number of elements, which will be counted with graphical chart; - min_quantity (default
1
) – a minimal appropriate value at with element will be added to the sheet; - ignore_case (default
False
) – with this argument asTrue
lower- and upper- case symbols will be united into a single element. WithFalse
– will be counted separately.Keyword-only
.
Main result info – number of unique entries, total count and average position (if exists) for each data type.
Top list of all analyzed symbols sorted by quantity. The next to it is also located the same one list, but with ignore-case. There is no need to create separate sheet, just use column of your choice.
Top list of symbol bigrams sorted by quantity with additional case insensitive
top-list.
Top list of analyzed words sorted by quantity. Word counting is always case insensitive, on the Analyze
stage.
Top list of analyzed word bigrams sorted by quantity.
2D sheet with all bigrams quantity. min_quantity
argument works here for sum of row/column values instead of each separated bigram.
Single call of all Result methods above. Order of the tuple arguments are the same as order of description above.
Please note – the last one (value for sheet_all_symbol_bigrams()) there is only in the min_quantities argument.
Default values as elsewhere: limits – (0,)*4; chart_limits – (20,)*4; min_quantities – (1,)*5.
Create symbols top-list as sheet_top_symbols()
, but only with symbols of your choice.
name
– keyword-only
Create symbols top-list as sheet_top_symbols()
, but only with base Latin symbols.
Create symbols top-list as sheet_top_symbols()
, but only with Russian Cyrillic symbols.
Create symbol bigrmas 2D sheet as sheet_all_symbol_bigrams()
, but only with symbols of your choice.
Order of symbols on the sheet will be the same as in the input argument.
name
– keyword-only
Create symbol bigrams 2D sheet as sheet_all_symbol_bigrams()
, but only with base Latin symbols.
Create symbol bigrams 2D sheet as sheet_all_symbol_bigrams()
, but only with Russian Cyrillic symbols.
Create cross-referenced sheet for all counted ye-yo words with their quantity and total misspells counter. Works only with analysis created with yo
argument as 1
or 2
.
- English analysis with EuroMatrixPlus/MultiUN English data set (3.1Gb .xml, 2.4*109 symbols, 379*106 words)
- Russian analysis with EuroMatrixPlus/MultiUN Russian data set (4.3Gb .xml, 2.2*109 symbols, 270*106 words)
- Russian analysis with OpenCorpora data set (528Mb .xml, 11.7*106 symbols, 1.6*106 words)