Yoshikoder import for dictionaries not working fully

[kbenoit]

(from http://stackoverflow.com/questions/43819992/quanteda-applying-yoshikoder-dictionary-with-multiple-levels)

Consider the file http://yoshikoder.sourceforge.net/code/yoshikoder/dictionaries/LaverGarryAJPS.ykd (from here). If you download this and try to import it, it only shows two keys, even though the dictionary contains many more.

> mydict <- dictionary(file = "~/Desktop/LaverGarryAJPS.ykd")
> mydict
Dictionary object with 2 key entries.
- Rural:
  - badgers, countryside, feed, hens, livestock, meadows, wildlife, agricultur*, bird*, farm*, fish*, forest*, horse*, landscape*, lane*, village*
- Urban:
  - town*

[conjugateprior]

If you're happy to change the package's dependency from XML to xml2 (it's massively better, faster, and apparently I'm the only one who uses the XML package anyway), then a fix is to collapse all the hierarchical structure of the dictionary like this:

read_dict_yoshikoder <- function(path){
  doc <- xml2::read_xml(path)
  pats <- xml2::xml_find_all(doc, ".//pnode")
  pnode_names <- xml2::xml_attr(pats, "name")  
  get_pnode_path <- function(pn, sep = ">") {
    pars <- xml2::xml_attr(xml2::xml_parents(pn), "name")
    paste0(rev(na.omit(pars)), collapse = sep)
  }
  pnode_paths <- lapply(pats, get_pnode_path)
  lst <- split(pnode_names, unlist(pnode_paths))
  dictionary(lst)
}

So

read_dict_yoshikoder("~/Documents/laver-garry-ajps.ykd")

gives

Dictionary object with 19 key entries.
 - Laver and Garry>Culture>High: art, artistic, dance, galler*, museum*, music*, opera*, theatre*
 - Laver and Garry>Culture>Popular: media
 - Laver and Garry>Culture>Sport: angler*
 - Laver and Garry>Environment>Con: produc*
 - Laver and Garry>Environment>Pro: car, catalytic, congestion, energy-saving, fur, green, husbanded, opencast, ozone, planet, population, re-use, toxic, warming, chemical*, chimney*, clean*, cyclist*, deplet*, ecolog*, emission*, environment*, habitat*, hedgerow*, litter*, open-cast*, recycl*, re-cycl*
 - Laver and Garry>Groups>Ethnic: race, asian*, buddhist*, ethnic*, raci*
 - Laver and Garry>Groups>Women: girls, woman, women
 - Laver and Garry>Institutions>Conservative: authority, legitimate, moratorium, whitehall, continu*, disrupt*, inspect*, jurisdiction*, manag*, rul*, strike*
 - Laver and Garry>Institutions>Neutral: assembly, headquarters, office, offices, official, opposition, queen, voting, westminster, administr*, advis*, agenc*, amalgamat*, appoint*, chair*, commission*, committee*, constituen*, council*, department*, directorate*, executive*, legislat*, mechanism*, minister*, operat*, organisation*, parliament*, presiden*, procedur*, process*, regist*, scheme*, secretariat*, sovereign*, subcommittee*, tribunal*, vote*
 - Laver and Garry>Institutions>Radical: abolition, accountable, answerable, scrap, consult*, corrupt*, democratic*, elect*, implement*, modern*, monitor*, rebuild*, reexamine*, reform*, re-organi*, repeal*, replace*, representat*, scandal*, scrap*, scrutin*, transform*, voice*
 - Laver and Garry>Law and Order>Conservative: assaults, bail, court, courts, dealing, delinquen*, deter, disorder, fine, fines, firmness, police, policemen, policing, probation, prosecution, re-offend, ruc, sentence*, shop-lifting, squatting, uniformed, unlawful, victim*, burglar*, constab*, convict*, custod*, deter*, drug*, force*, fraud*, guard*, hooligan*, illegal*, intimidat*, joy-ride*, lawless*, magistrat*, offence*, officer*, penal*, prison*, punish*, seiz*, terror*, theft*, thug*, tough*, trafficker*, vandal*, vigilan*
 - Laver and Garry>Law and Order>Liberal: harassment, non-custodial
 - Laver and Garry>Rural: badgers, countryside, feed, hens, livestock, meadows, wildlife, agricultur*, bird*, farm*, fish*, forest*, horse*, landscape*, lane*, village*
 - Laver and Garry>State in Economy>Con: assets, autonomy, bid, bidders, bidding, confidence, confiscatory, controlled, controlling, controls, corporate, deregulating, expensive, fund-holding, initiative, intrusive, monetary, money, private, privately, privatisations, privatised, privatising, profitable, risk, risks, savings, shares, sponsorship, taxable, taxes, tax-free, trading, value, barrier*, burden*, charit*, choice*, compet*, constrain*, contracting*, contractor*, corporation*, dismantl*, entrepreneur*, flexib*, franchise*, fundhold*, homestead*, investor*, liberali*, market*, own*, produce*, regulat*, retail*, sell*, simplif*, spend*, thrift*, volunt*, voucher*
 - Laver and Garry>State in Economy>Neut: accountant, accounting, accounts, bargaining, electricity, fee, fees, import, imports, jobs, opportunity, performance, productivity, settlement, software, supply, trade, welfare, advert*, airline*, airport*, audit*, bank*, breadwinner*, budget*, buy*, cartel*, cash*, charge*, chemical*, commerce*, compensat*, consum*, cost*, credit*, customer*, debt*, deficit*, dwelling*, earn*, econ*, estate*, export*, financ*, hous*, industr*, lease*, loan*, manufactur*, mortgage*, negotiat*, partnership*, passenger*, pay*, port*, profession*, purchas*, railway*, rebate*, recession*, research*, revenue*, salar*, sell*, supplier*, telecom*, telephon*, tenan*, touris*, train*, wage*, work*
 - Laver and Garry>State in Economy>Pro: accommodation, age, ambulance, assist, benefit, care, class, classes, clinics, deprivation, disabilities, disadvantaged, elderly, establish, hardship, hunger, invest, investing, investment, patients, pension, poor, poorer, poorest, poverty, school, transport, vulnerable, carer*, child*, collective*, contribution*, cooperative*, co-operative*, educat*, equal*, fair*, guarantee*, health*, homeless*, hospital*, inequal*, means-test*, nurse*, rehouse*, re-house*, teach*, underfund*, unemploy*, widow*
 - Laver and Garry>Urban: town*
 - Laver and Garry>Values>Conservative: defend, defended, defending, discipline, glories, glorious, grammar, heritage, integrity, maintain, majesty, marriage, past, pride, probity, professionalism, proud, histor*, honour*, immigra*, inherit*, jubilee*, leader*, obscen*, pornograph*, preserv*, principl*, punctual*, recapture*, reliab*, threat*, tradition*
 - Laver and Garry>Values>Liberal: innocent, inter-racial, rights, cruel*, discriminat*, human*, injustice*, minorit*, repressi*, sex*

Does that work for you all?

[conjugateprior]

I forget, does dictionary do nested categories?

[kbenoit]

Yes it does do nested categories, see ?tokens_lookup. We want to preserve the nested list structure. The bug in this issue is that somehow, we are not reading the whole nested structure - or even the whole top-level structure!

If you want to take a stab at solving this, go ahead - happy to shift to xml2. All of the calls to XML are in dictionaries.R.