dictionary(file = "some_wordstat_formatted_dictionary.cat") not robust

[kbenoit]

Some odd behaviours depending on whether a file is created explicitly using the Wordstat formatting rules, or typed into a text editor which should be following those same rules.

This has to to do with the readWStatDict() in dictionaries.R.

Related to #458.

dicttxt <-  "SOUTH\n\tBW\n\t\tBotswana (1)\n\t\tGaborone (1)\n\tLS\n\t\tLesotho (1)\n\t\tLesothonian* (1)\n\t\tMaseru (1)\n\tNA\n\t\tNamibia (1)"
cat(dicttxt)
cat(dicttxt, file = (f <- tempfile(fileext = ".cat")))
# SOUTH
#       BW
#               Botswana (1)
#               Gaborone (1)
#       LS
#               Lesotho (1)
#               Lesothonian* (1)
#               Maseru (1)
#       NA
#               Namibia (1)

# works fine from above
dictionary(file = f)
# Dictionary object with 3 key entries.
#  - SOUTH.BW: botswana, gaborone
#  - SOUTH.LS: lesotho, lesothonian*, maseru
#  - SOUTH.NA: namibia

# does not work fine from this file
dictionary(file = "../data/dictionaries/issue-459.cat")
# Dictionary object with 3 key entries.
#  -    .BW: botswana, gaborone
#  -    .LS: lesotho, lesothonian*, maseru
#  -    .NA: namibia

# compare: (not shown)
file.show(f)
file.show("../data/dictionaries/issue-459.cat")

[koheiw]

I find none of the format currently supported by the dictionary() is really up to complex tasks and robust (not quotation, not comments etc.). It is important to provide interfaces to existing formats, but creating a new dictionary in a deficient format does not make sense at all. My suggestion is to support YAML, which is a well defined format available through YAML package. I also wrote a new fatten dicitonary(), with the levels argument.

str <- "
AFR:
 SOUTH:
  BW: [Botswana, Botswanan*, Gaborone]
  LS: [Lesotho, Lesothonian*, Maseru]
  NA: [Namibia, Namibian*, Windhoek]
  ZA: [South Africa, S African, SA, South African*, S African*, Cape Town, Johannesburg]
  SZ: [Swaziland, Swazi*, Lobamba, Mbabane]
"
dict <- yaml::yaml.load(str)

dictionary_flatten <- function(tree, levels = 1:100, level = 1, key_tree = '', dict = list()) {
    #cat("-------------------\n")
    #cat("level:", level, "\n")
    
    if (is.null(names(tree))) {
        stop("dictionary elements must be named")
    }
    if (any(names(tree) == "")) {
        unnamed <- tree[which(names(tree) == "")]
        stop("unnamed dictionary entry: ", unnamed)
    }
    for (name in names(tree)) {
        branch <- tree[[name]]
        if (level %in% levels) {    
            if (key_tree != '') {
                key <- paste(key_tree, name, sep = '.')
            } else {
                key <- name
            }
        } else {
            key <- key_tree
        }
        #cat("key:", key, "\n")
        #cat("key_tree:", key_tree, "\n")
        if (is.list(branch)) {
            #cat("List:\n")
            #print(branch)
            dict <- dictionary_flatten(branch, levels, level + 1, key, dict)
        } else {
            #cat("Vector:\n")
            #print(branch)
            dict[[key]] <- c(dict[[key]], branch)
        }
    }
    return(dict)
}

print(dictionary_flatten(dict, 1))
print(dictionary_flatten(dict, 1:3))
print(dictionary_flatten(dict, 2:3))
print(dictionary_flatten(dict, 3))

[koheiw]

We could convert wordstat, LIWC, etc. dictionaries to YAMLs with regular expression, and ask yaml::yaml.load() to parse them.

[kbenoit]

If that's more robust, fine - it's just an under-the-hood method of parsing foreign formats. And of course it would be nice to suggest YAML as a natural way of authoring plain-text dictionaries.

The LIWC is the gnarliest of the foreign formats, and probably not worth trying to read this way. But Wordstat would work fine in this fashion.

Yoshikoder is already XML so no real need to convert to "yet another" markup language.

[kbenoit]

From a user-facing standpoint, we'd simply be adding one more import format to the existing list, it would not change the function.

The issue of handling levels is another issue. I'd suggest we keep all levels in a dictionary but add a levels argument to the *_lookup functions.