Support hierarchical LIWC

[leezu]

read_dict_liwc does not read the hierarchical LIWC format correctly (i.e. LIWC2015_English.dic).

Consider this partial excerpt from the categories:

30	affect (Affect)
	31	posemo (Positive Emotions)
	32	negemo (Negative Emotions)
		33	anx (Anx)
		34	anger (Anger)
		35	sad (Sad)

Instead of the posemo and negemo columns the final dataframe only contains a Emotions) column.

[koheiw]

Thank you for reporting the issue. We were not aware that there is a new LIWC format, and developed read_dict_liwc only for the 2007 format. Once we obtained the new dictionary, we will upgrade the importer.

[koheiw]

@kbenoit , I am working on this issue, but I do not understand this. Can you explain?
https://github.com/kbenoit/quanteda/blob/88009a4b3ffed07d32997c7282c0ed6ae3d06bc8/R/dictionaries.R#L275-L287

[kbenoit]

I'm not sure what that means now! I could probably figure it out later this afternoon, when my meetings are done, by getting back into that code.

[koheiw]

The conversion of dictionary to list on line 284 destroyer the structure that allows a key to have both values and sub-keys. For example, in the LIWC 2015 format, 'past' has a value liked as well as sub-keys 'cogmech' and 'insight'. Internally, read_dict_liwc() constructs a list of lists, but as.list() coverts the the element to a list of lists and a character vector named '__'. Dictionary should always be a list of lists, always.

%
11  verb
12  misc
	121 whatever
	122	family
	123	friend
	124	humans
	125	affect
	126	posemo
13  past
	131	cogmech
	132	insight
		133	cause
		134	discrep
		135	tentat
else 
	140 whatever2
	253	time
	464	filler
nothing
%
kin	121	122
kind	<of>131/125	<of>135/126
kinda	131	135
light	140
like	(02 134)125/464	(02 134)126	(02 134)126	253
likeab*	125	126
liked	11	13	125	126 721

dictionary(file = "../data/dictionaries/liwc_hierarchical.dic")

# Dictionary object with 4 primary key entries and 3 nested levels.
# - verb:
#     - liked
# - misc:
#     - whatever:
#     - kin
# - family:
#     - kin
# - affect:
#     - likeab*, liked
# - posemo:
#     - like, likeab*, liked
# - past:
#     - cogmech:
#     - kinda
# - insight:
#     - tentat:
#     - kinda
# - __:
#     - liked
# - else:
#         - whatever2:
#     - light
# - time:
#     - like

[kbenoit]

OK, I see the merit of this approach. The read* functions currently are a mix of list of characters and list of lists, and there is a function that converts. I think that when I fixed a bug in the last major changes to the dictionary functions, I undid some of your "always list of lists approach." Happy for you to go back to the list of lists, as long as the tests pass.

But we should have:

• dictionary() takes a list of characters as input, not a list of lists
• as.list.dictionary() returns (for a one-level list anyway) a list of characters.

So the input to the constructor, and the output from the coercion back to list, will not have a list as the lowest level, but instead a character vector as the lowest level.

[koheiw]

@leezu please try the latest master.