Lucene index missing

[LukeButters]

I am trying to run

./run @mode=freebase @domain=free917 @sparqlserver=localhost:3093 @cacheserver=localhost:4000 @data=1

It seems that the lucene index is missing it complained sempre/lib/lucene/4.4/free917 was missing, I created the folder hoping it was just expecting the folder to exist and it would write files to that directory. It seems that is also not the case. I now get:

SparqlExecutor.execute: (!fb:finance.currency.currency_code fb:en.canadian_dollar) 
            Parser.parse: parse {
              Constructing Searcher {
                Opening index dir: lib/lucene/4.4/free917/
                ERROR: Composition failed: rule = $Entity -> $TokenSpan (LexiconFn entity exact), children = [(derivation (formula (string what)) (type fb:type.text))]
java.lang.RuntimeException: org.apache.lucene.index.IndexNotFoundException: no segments* file found in org.apache.lucene.store.MMapDirectory@/home/funnelback/sempre/lib/lucene/4.4/free917 lockFactory=org.apache.lucene.store.NativeFSLockFactory@28177e78: files: []
        at edu.stanford.nlp.sempre.freebase.LexiconFn.call(LexiconFn.java:237)
        at edu.stanford.nlp.sempre.BeamParserState.applyRule(BeamParser.java:142)
        at edu.stanford.nlp.sempre.BeamParserState.applyCatUnaryRules(BeamParser.java:193)
        at edu.stanford.nlp.sempre.BeamParserState.build(BeamParser.java:126)
        at edu.stanford.nlp.sempre.BeamParserState.infer(BeamParser.java:98)
        at edu.stanford.nlp.sempre.Parser.parse(Parser.java:159)
        at edu.stanford.nlp.sempre.Learner.parseExample(Learner.java:252)
        at edu.stanford.nlp.sempre.Learner.processExamples(Learner.java:169)
        at edu.stanford.nlp.sempre.Learner.learn(Learner.java:111)
        at edu.stanford.nlp.sempre.Learner.learn(Learner.java:82)
        at edu.stanford.nlp.sempre.Main.run(Main.java:23)
        at fig.exec.Execution.runWithObjArray(Execution.java:337)
        at fig.exec.Execution.run(Execution.java:325)
        at edu.stanford.nlp.sempre.Main.main(Main.java:38)
Caused by: org.apache.lucene.index.IndexNotFoundException: no segments* file found in org.apache.lucene.store.MMapDirectory@/home/funnelback/sempre/lib/lucene/4.4/free917 lockFactory=org.apache.lucene.store.NativeFSLockFactory@28177e78: files: []
        at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:770)
        at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:52)
        at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:66)
        at edu.stanford.nlp.sempre.freebase.index.FbEntitySearcher.<init>(FbEntitySearcher.java:45)
        at edu.stanford.nlp.sempre.freebase.EntityLexicon.lookupEntries(EntityLexicon.java:69)
        at edu.stanford.nlp.sempre.freebase.Lexicon.lookupEntities(Lexicon.java:65)
        at edu.stanford.nlp.sempre.freebase.LexiconFn.call(LexiconFn.java:204)
        ... 13 more
                ERROR: java.lang.RuntimeException: java.lang.RuntimeException: org.apache.lucene.index.IndexNotFoundException: no segments* file found in org.apache.lucene.store.MMapDirectory@/home/funnelback/sempre/lib/lucene/4.4/free917 lockFactory=org.apache.lucene.store.NativeFSLockFactory@28177e78: files: []:
edu.stanford.nlp.sempre.BeamParserState.applyRule(BeamParser.java:165)
edu.stanford.nlp.sempre.BeamParserState.applyCatUnaryRules(BeamParser.java:193)
edu.stanford.nlp.sempre.BeamParserState.build(BeamParser.java:126)
edu.stanford.nlp.sempre.BeamParserState.infer(BeamParser.java:98)
edu.stanford.nlp.sempre.Parser.parse(Parser.java:159)
edu.stanford.nlp.sempre.Learner.parseExample(Learner.java:252)
edu.stanford.nlp.sempre.Learner.processExamples(Learner.java:169)
edu.stanford.nlp.sempre.Learner.learn(Learner.java:111)
edu.stanford.nlp.sempre.Learner.learn(Learner.java:82)
edu.stanford.nlp.sempre.Main.run(Main.java:23)
fig.exec.Execution.runWithObjArray(Execution.java:337)
fig.exec.Execution.run(Execution.java:325)
edu.stanford.nlp.sempre.Main.main(Main.java:38)
                ERROR: Caused by java.lang.RuntimeException: org.apache.lucene.index.IndexNotFoundException: no segments* file found in org.apache.lucene.store.MMapDirectory@/home/funnelback/sempre/lib/lucene/4.4/free917 lockFactory=org.apache.lucene.store.NativeFSLockFactory@28177e78: files: []:
edu.stanford.nlp.sempre.freebase.LexiconFn.call(LexiconFn.java:237)
edu.stanford.nlp.sempre.BeamParserState.applyRule(BeamParser.java:142)
edu.stanford.nlp.sempre.BeamParserState.applyCatUnaryRules(BeamParser.java:193)
edu.stanford.nlp.sempre.BeamParserState.build(BeamParser.java:126)
edu.stanford.nlp.sempre.BeamParserState.infer(BeamParser.java:98)
edu.stanford.nlp.sempre.Parser.parse(Parser.java:159)
edu.stanford.nlp.sempre.Learner.parseExample(Learner.java:252)
edu.stanford.nlp.sempre.Learner.processExamples(Learner.java:169)
edu.stanford.nlp.sempre.Learner.learn(Learner.java:111)
edu.stanford.nlp.sempre.Learner.learn(Learner.java:82)
edu.stanford.nlp.sempre.Main.run(Main.java:23)
fig.exec.Execution.runWithObjArray(Execution.java:337)
fig.exec.Execution.run(Execution.java:325)
edu.stanford.nlp.sempre.Main.main(Main.java:38)
                Execution directory: state/execs/6.exec

[ppasupat]

Hello. Could you try running

./pull-dependencies freebase

and then run the code again?

[LukeButters]

re-ran the pull and it seems lucene continues to complain about missing segments
ERROR: Caused by java.lang.RuntimeException: org.apache.lucene.index.IndexNotFoundException: no segments* file found in org.apache.lucene.store.MMapDirectory@/home/luke/sempre/lib/lucene/4.4/free917 lockFactory=org.apache.lucene.store.NativeFSLockFactory@28177e78: files: []:

[ppasupat]

It looks like the dependencies for the Free917 experiments got lost from the dependencies list.
For now, you can download the required files at

• http://nlp.stanford.edu/software/sempre/release-emnlp2013/lib/lucene/4.4/free917.tar.bz2

Extract it into lib/lucene/4.4/:

lib
'-- lucene
    '-- 4.4
        '-- free917
            '-- segments.gen, segments_1, ...

[LukeButters]

Ah that seems to have helped!
Although now it seems for w/e it is doing when it starts it is timing out on all queries:

              SparqlExecutor.execute: (count (and (fb:type.object.type fb:location.location) (!fb:location.country.capital fb:en.united_states_of_america))) {
                Error: TIMEOUT
              } [1m0s, cum. 37m56s]

perhaps my server is too slow.

[LukeButters]

Is it possible to be left with an interactive prompt (like in the tutorial) with the free917 data?

[ghost]

In case this helps anyone, I was getting a lib/lucene/4.4 directory doesn't exist while training on the webquestions dataset.

I switched to v1.0-fix branch and did:

./download-dependencies emnlp2013

copied the resulting lib/lucene folder into the master branch and ran it again and it works.