Skip to content

corpus_segment(x, what = "tags") is broken when a file starts with \n##TAG #618

Description

@kbenoit
testCorpus <- corpus(c("\n##INTRO This is the introduction.
                        ##DOC1 This is the first document.  Second sentence in Doc 1.
                        ##DOC3 Third document starts here.  End of third document.",
                       "##INTRO Document ##NUMBER Two starts before ##NUMBER Three."))
testCorpusSeg <- corpus_segment(testCorpus, "tags")
summary(testCorpusSeg)
## Corpus consisting of 7 documents.
## 
##     Text Types Tokens Sentences      tag
##  text1.1     0      0         0  ##INTRO
##  text1.2     5      5         1   ##DOC1
##  text1.3    11     12         2   ##DOC3
##  text1.4     8     10         2  ##INTRO
##  text2.1     1      1         1 ##NUMBER
##  text2.2     3      3         1 ##NUMBER
##  text2.3     2      2         1     <NA>
##     
## Source:  /Users/kbenoit/Dropbox (Personal)/GitHub/quanteda/* on x86_64 by kbenoit
## Created: Sat Mar 25 16:50:47 2017
## Notes:   corpus_segment(corpus_segment.corpus)corpus_segment(testCorpus)corpus_segment(tags)

Clearly, the first document (tag) should not be zero tokens.

Metadata

Metadata

Assignees

Labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions