Skip to content

Commit

Permalink
* with r5137 robotParser didn't work at all -> fix
Browse files Browse the repository at this point in the history
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5139 6c8d7289-2bf4-0310-a012-ef5d649a1542
  • Loading branch information
f1ori committed Sep 12, 2008
1 parent 3ded1ef commit bea6c13
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion source/de/anomic/crawler/robotsParser.java
Expand Up @@ -109,7 +109,7 @@ private void parse(final BufferedReader reader) {
try {
while ((line = reader.readLine()) != null) {
// replacing all tabs with spaces
line = line.replaceAll("\t"," ").replaceAll(":"," ").trim();
line = line.replaceAll("\t"," ").trim();
lineUpper = line.toUpperCase();

if (line.length() == 0) {
Expand Down

0 comments on commit bea6c13

Please sign in to comment.