object constructors should work on the class they construct

[kbenoit]

dfm() already does this, for instance. We should add:

• dfm.dfm() needs thorough testing to make sure the options work
• fcm() needs implementation and testing
• tokens.tokens() needs implementation and testing
• corpus.corpus() (implemented in Add constructors for objects of their own class #880)
• dictionary() (implemented in Add constructors for objects of their own class #880)

Note that the key here is not simply to make the constructor work, but also for all of the options to work the way they should, to change the input object. We will need to test each of these.

[kbenoit]

@koheiw Here's a use case for tokens.tokens(): Right now, the remove_* options don't work in dfm.tokens(x), because the x is already tokenised and therefore (e.g.) remove_punct is never passed to tokens(). If we always called tokens() as part of dfm(), however, then this would be more consistent behaviour in dfm() and solve this problem generally (where the "problem" is that this not only did not remove punctuation, but also failed to issue any warning).

toks <- tokens("This, is punctuation.")
dfm(toks, remove_punct = TRUE)
## Document-feature matrix of: 1 document, 5 features (0% sparse).
## 1 x 5 sparse Matrix of class "dfmSparse"
##        features
## docs    this , is punctuation .
##   text1    1 1  1           1 1

So I'm suggesting that we want this to work:

tokens(toks, remove_punct = TRUE)

and (hopefully, obviously) for this to be TRUE:

identical(toks, tokens(toks))

[koheiw]

remove_punct = TRUE is done by stri_split_boundaries(skip_word_none = TRUE), so we probably need to change how tokenization is done to make identical(toks, tokens(toks)).