-
Notifications
You must be signed in to change notification settings - Fork 33
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error: Character string limit #24
Comments
This is probably because in the tagging, the output is put into 1 string (the result of
The solution is just to split the data in chunks and apply the annotation, something like this?
It seems to me that in that case, the final conllu parsed result will not be larger than 2^31-1 bytes) |
Thank you for replying, your suggested solution works. In addition, using
|
Good to know. Feel free to close the issue. |
I am trying to tokenise, lemmatise and parts-of-speech tag a large corpus of English texts. There are approximately 160,000 texts in the corpus, totaling approximately 46 million tokens, which means that on average the individual texts are relatively short (approx. 290 tokens). Following the example of the
![corpus_sample](https://user-images.githubusercontent.com/13680208/39750087-9d00c466-52b4-11e8-9ee5-df6d915ddea3.png)
brussels_reviews
dataset, the corpus is stored in a data table with individual raw texts in one column (see attached screenshot of corpus sample).As usual, I try to parse the corpus by calling
This works like a charm for a Polish corpus of approximately 13 million tokens. However, for English I get the following error message:
I am puzzled by this error, because the longest of all texts in the corpus is only 32,458 characters long, as evidenced by the column
textlen
from the corpus data table. The sum of all characters across all texts is 274,025,244, which is less than the limit stipulated by the error message. As a further sanity check, I splitted the whole corpus on white spaces bycorpus.dt[, .(word = unlist(strsplit(text, "[[:space:]]")), doc_nr = .GRP), by = .(doc_id)]
in order to assess the length of the longest token in the corpus, and this method revealed that the longest token in the corpus (obtained by the simple white-space-splitting procedure) is only 114 characters long. Therefore, I have no idea what could have caused the error.Do you have any idea or suggestion as to what could have gone wrong? As mentioned before, everything worked perfectly fine with a smaller Polish corpus. I will appreciate any comment on this regard, because otherwise your your UDPipe implementation does everything I need without any problems.
The text was updated successfully, but these errors were encountered: