Indexation is dammed slow with large data

[jmrenouard]

Hello,

We have face issues around the slow indexation process.

We have decided to rewrite 2 methods in FilesystemStore :
rebuildAllIndexes and rebuild(String name)

The indexation performance are now closed to solr/lucene.

I put the code on the issue sorry for no patch
the getDateTime method is just for having some timing information.

private String getDateTime() {
    DateFormat dateFormat = new SimpleDateFormat("yyyy/MM/dd HH:mm:ss");
    Date date = new Date();
    return dateFormat.format(date);
}
public void rebuildAllIndexes() throws Exception {
    Logger.info("Rebuild All indexes at "+getDateTime());
    stop();
    File fl = new File(DATA_PATH);
    FileUtils.deleteDirectory(fl);
    fl.mkdirs();
    List<ApplicationClass> classes = Play.classes
            .getAnnotatedClasses(Indexed.class);
    for (ApplicationClass applicationClass : classes) {
        rebuild(applicationClass.javaClass.getName());
    }

    Logger.info("Rebuild indexes finished at "+getDateTime());
}

public void rebuild(String name) {
    String id = UUID.randomUUID().toString();
    File oldFolder = new File(DATA_PATH, name);
    File newFolder = new File(DATA_PATH, name + id);
    Class cl = Play.classes.getApplicationClass(name).javaClass;
    List<JPABase> objects = JPA
            .em()
            .createQuery("select e from " + cl.getCanonicalName() + " as e")
            .getResultList();
    String index = cl.getName() + id;
    IndexWriter indexWriter = getIndexWriter(index);
    // FIXME ensure no other read/writes in here.
    try {
        for (JPABase jpaBase : objects) {
            Document document = ConvertionUtils.toDocument(jpaBase);
            if (document == null)
                return;
            indexWriter.addDocument(document);
        }

        getIndexWriter(index).flush();
        dirtyReader(index);
        getIndexSearcher(name).close();
        indexSearchers.remove(name);
        getIndexWriter(name).close();
        indexWriters.remove(name);
        FileUtils.deleteDirectory(oldFolder);
        newFolder.renameTo(oldFolder);
    } catch (IOException e) {
        throw new UnexpectedException(e);
    } catch (Exception e) {
        throw new UnexpectedException(e);
    }
}

[jfp]

Hi,

Thank's for sharing the code, I'll take a look. (You were running with play.search.sync=false ?).

Regards

[jmrenouard]

yes

play.search.sync=false;

for 45000 objects we need more than 5 minutes.
Now it takes 5 to 6 seconds.

10 seconds are needed with information printed about each document indexed.

Best Regards,
Jean-Marie Renouard

[jfp]

Hi,

It's integrated in e69fd1e

Thanks again for sharing !