-
Notifications
You must be signed in to change notification settings - Fork 69
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Problematic tokens from tokenizer? #6
Comments
About the |
I meet the similar problem with you. My error message is |
please delete the .DS_Store file from the dataset directory. |
I'm hoping this solved the problem. If not, please open another issue. |
Some tokens from some of the tokenized files seem problematic for SourcererCC.
Here is an example of
stderr
when the indexing fails, the contents is e.g. weird whitespaces or chars like||
:While indexing, there is a lot of
EXCEPTION CAUGHT
messages coming from caughtArrayIndexOutOfBoundsException
s in CloneHelper.java. I'm not sure if it's a problem or not.Also, while searching I am getting a lot of
ERROR: more that one doc found. some error here.
messages.Maybe these problems are some small things in the tokenization process, you have any ideas what it might be? For now, I will update the handling of weird whitespaces and see how it helps.
The text was updated successfully, but these errors were encountered: