Skip to content

Improve corpus.Corpus() to read document data #849

Description

@kbenoit

Simple corpus objects just have a character vector in each list element's content value, e.g.

> data(crude, package = "tm")
> str(crude[[1]])
List of 2
 $ content: chr "Diamond Shamrock Corp said that\neffective today it had cut its contract prices for crude oil by\n1.50 dlrs a b"| __truncated__
 $ meta   :List of 15
  ..$ author       : chr(0) 
  ..$ datetimestamp: POSIXlt[1:1], format: "1987-02-26 17:00:56"
  ..$ description  : chr ""
  ..$ heading      : chr "DIAMOND SHAMROCK (DIA) CUTS CRUDE PRICES"
  ..$ id           : chr "127"
  ..$ language     : chr "en"
  ..$ origin       : chr "Reuters-21578 XML"
  ..$ topics       : chr "YES"
  ..$ lewissplit   : chr "TRAIN"
  ..$ cgisplit     : chr "TRAINING-SET"
  ..$ oldid        : chr "5670"
  ..$ places       : chr "usa"
  ..$ people       : chr(0) 
  ..$ orgs         : chr(0) 
  ..$ exchanges    : chr(0) 
  ..- attr(*, "class")= chr "TextDocumentMeta"
 - attr(*, "class")= chr [1:2] "PlainTextDocument" "TextDocument"

But content may also be a data.frame, with the columns text and any other data:

> str(corpusGER[[1]])
List of 2
 $ content:'data.frame':	5675 obs. of  2 variables:
  ..$ text    : chr [1:5675] "wir wenden uns mit diesem Programm an Sie." "Wir wollen Sie bei der Bundestagswahl am 22. September für eine andere, für eine bessere Politik gewinnen." "Wir wenden uns mit unserem Programm an alle, die meinen, dass wir in unserer Gesellschaft jetzt einiges verände"| __truncated__ "Wenn wir die Klimakatastrophe so weit wie möglich aufhalten und die Energiewende verwirklichen," ...
  ..$ cmp_code: chr [1:5675] "305" "305" "305" "501" ...
 $ meta   :List of 14
  ..$ manifesto_id               : chr "41113_201309"
  ..$ party                      : num 41113
  ..$ date                       : num 201309
  ..$ language                   : chr "german"
  ..$ source                     : chr "MARPOR"
  ..$ has_eu_code                : logi FALSE
  ..$ is_primary_doc             : logi TRUE
  ..$ may_contradict_core_dataset: logi FALSE
  ..$ md5sum_text                : chr "c5ce2ef21338fafb114c6ea352422921"
  ..$ url_original               : chr "/down/originals/41113_2013.pdf"
  ..$ md5sum_original            : chr "CURRENTLY_UNAVAILABLE"
  ..$ annotations                : logi TRUE
  ..$ handbook                   : chr "4"
  ..$ id                         : chr "41113_201309"
  ..- attr(*, "row.names")= int 1
  ..- attr(*, "class")= chr [1:2] "ManifestoDocumentMeta" "TextDocumentMeta"

(Example comes from downloaded manifesto data using the manifestoR package.)

Goal: We want to include the additional variables as docvars, while getting the text as appropriate.

Metadata

Metadata

Assignees

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions