Comments are not removed in code_tokens

[v587su]

This is a data from java_test_0.jsonl file. There are still comments in code_tokens. I highlight them using black.

"repo": "ReactiveX/RxJava", "path": "src/main/java/io/reactivex/Observable.java", "func_name": "Observable.skipLast", "original_string": "@CheckReturnValue\n @SchedulerSupport(SchedulerSupport.CUSTOM)\n public final Observable skipLast(long time, TimeUnit unit, Scheduler scheduler, boolean delayError, int bufferSize) {\n ObjectHelper.requireNonNull(unit, "unit is null");\n ObjectHelper.requireNonNull(scheduler, "scheduler is null");\n ObjectHelper.verifyPositive(bufferSize, "bufferSize");\n // the internal buffer holds pairs of (timestamp, value) so double the default buffer size\n int s = bufferSize << 1;\n return RxJavaPlugins.onAssembly(new ObservableSkipLastTimed(this, time, unit, scheduler, s, delayError));\n }", "language": "java", "code": "@CheckReturnValue\n @SchedulerSupport(SchedulerSupport.CUSTOM)\n public final Observable skipLast(long time, TimeUnit unit, Scheduler scheduler, boolean delayError, int bufferSize) {\n ObjectHelper.requireNonNull(unit, "unit is null");\n ObjectHelper.requireNonNull(scheduler, "scheduler is null");\n ObjectHelper.verifyPositive(bufferSize, "bufferSize");\n // the internal buffer holds pairs of (timestamp, value) so double the default buffer size\n int s = bufferSize << 1;\n return RxJavaPlugins.onAssembly(new ObservableSkipLastTimed(this, time, unit, scheduler, s, delayError));\n }", "code_tokens": ["@", "CheckReturnValue", "@", "SchedulerSupport", "(", "SchedulerSupport", ".", "CUSTOM", ")", "public", "final", "Observable", "<", "T", ">", "skipLast", "(", "long", "time", ",", "TimeUnit", "unit", ",", "Scheduler", "scheduler", ",", "boolean", "delayError", ",", "int", "bufferSize", ")", "{", "ObjectHelper", ".", "requireNonNull", "(", "unit", ",", ""unit is null"", ")", ";", "ObjectHelper", ".", "requireNonNull", "(", "scheduler", ",", ""scheduler is null"", ")", ";", "ObjectHelper", ".", "verifyPositive", "(", "bufferSize", ",", ""bufferSize"", ")", ";", "// the internal buffer holds pairs of (timestamp, value) so double the default buffer size", "int", "s", "=", "bufferSize", "<<", "1", ";", "return", "RxJavaPlugins", ".", "onAssembly", "(", "new", "ObservableSkipLastTimed", "<", "T", ">", "(", "this", ",", "time", ",", "unit", ",", "scheduler", ",", "s", ",", "delayError", ")", ")", ";", "}"]

[mallamanis]

Hi @v587su we do remove the documentation comment (JavaDoc) but not any other comments as these could still be useful for code search. Feel free to manually remove them, if you feel that this would help.