Incredibly bad startup performance caused by loading shell-history file with long lines. #83
Description
The code of JLineShell.filterLogEntry()
is performing incredibly bad if the history file spring-shell.log contains very long lines.
To give you an example:
We have a history file containing 87 lines and a size of about 10MB, i.e. some of the contained lines are really long.
filterLogEntry() takes 6.5 minutes to read that file during the start of the application.
The current code looks like this:
private String[] filterLogEntry() {
ArrayList<String> entries = new ArrayList<String>();
ReversedLinesFileReader reversedReader = null;
try {
reversedReader = new ReversedLinesFileReader(new File(getHistoryFileName()), 4096, Charset.forName("UTF-8"));
int size = 0;
String line = null;
while ((line = reversedReader.readLine()) != null) {
if (!line.startsWith("//")) {
size++;
if (size > historySize) {
break;
}
else {
entries.add(line);
}
}
}
}
catch (IOException e) {
logger.warning("read history file failed. Reason:" + e.getMessage());
}
finally {
closeReversedReader(reversedReader);
}
Collections.reverse(entries);
return entries.toArray(new String[0]);
}ReversedLinesFileReader is the culprit here since it performs very very badly if the length of a line is larger than the buffer size.
Straightforward replacement suggestion:
private String[] filterLogEntryNew() {
try {
List<String> lines = IOUtils.readLines(new BufferedInputStream(new FileInputStream(historyFileName)), Charset.forName("UTF-8"));
Iterator<String> iter = lines.iterator();
while (iter.hasNext()) {
String line = iter.next();
if (line.startsWith("//")) {
iter.remove();
}
}
int totalSize = lines.size();
int size = Math.min(totalSize, historySize);
String[] result = new String[size];
int startIndex = totalSize - size;
for(int i=startIndex ; i<totalSize ; i++) {
result[i-startIndex] = lines.get(i);
}
return result;
} catch (IOException e) {
logger.warning("read history file failed. Reason:" + e.getMessage());
return new String[0];
}
}This code loads the 10MB spring-shell.log mentioned above in 150ms.
Feel free to use it if you like.
It's not possible to just replace the above method in an extending class because it's a private method, unfortunately.