Very large number of "Feature skipped" error messages

[GoogleCodeExporter]

When I run the following command I get many  "Feature skipped" error
messages. I previously populated the effcounts files with estimate-ngram -t
-wec for each model, and they look fine. What do these error messages mean?
I also tried this with CM, and got many of the Feature skipped messages,
and a "feature read from..." for each model.

interpolate-ngram -lm "model1.lm, model2.lm, model3.lm" -interpolation GLI
-op dev.txt -wl GLI.lm -if 
"log:sumhist:%s.effcounts,pow2:log1p:sumhist:%s.effcounts"


Thanks

Original issue reported on code.google.com by peppe...@gmail.com on 26 Feb 2009 at 11:27

[GoogleCodeExporter]

Wasn't sure how to chage the Type/Priority.

Original comment by peppe...@gmail.com on 26 Feb 2009 at 11:36

[GoogleCodeExporter]

Hi perpperbc,

I am having difficulty replicating the problem.  The feature skipped messages
indicate that there is an n-gram in the feature file that do not exist in the 
n-gram
model.  Currently, the only way I can think where this occurs is if the 
.effcounts
are computed for higher order n-grams (ex. 4) than the target n-gram model (ex. 
3). 
In this case, although you will get a lot of feature skipped warning messages, 
the
target LM should still be computed correctly.

Can you please confirm if this is the cause of the problem?  Regardless, I will
modify the warning messages to display more information and summarize the 
warnings
beyond 3 or so messages.

P.S. You can change the type and priority by clicking on the label itself.

Original comment by bojune...@gmail.com on 27 Feb 2009 at 5:03

• Changed state: Accepted

[GoogleCodeExporter]

Original comment by bojune...@gmail.com on 27 Feb 2009 at 5:03

[GoogleCodeExporter]

That seems like it would have been the problem, as I had forgotten to to use 
the the
-o 6 flag, and all the component models were 6-gram. Unfortunately, I got a 
segfault
when I added the flag.

Interpolating component LMs...
Tying parameters across n-gram order...
Segmentation fault

After removing, which became 5 ~1GB), I still got a very large number of 
"Feature
skipped."  error messages. There were also some "Feature value
pow2:log1p:sumhist:literal-all.effcounts exceed 20.0" after each effcounts file 
was
said to be loaded. It then segfaulted again, after having only read 3 of 5 
feature
count files.

I am running it one more time with only 4 models, and it has been streaming 
"Feature
skipped" messages for about half an hour ( though it hasn't crashed). I will 
update
if anything interesting happens with it.

I also ran a CM build with only two models "interpolate-ngram -lm "1.lm, 2.lm"
-interpolation CM -op dev.txt -wl CM.lm  -interpolation-features
"log:sumhist:%s.corp.txt" -o 6", 
got 22 Feature skipped messages, and it completed.

Are there any key variables I could have printed at the same time as the 
"Feature
skipped." message in order to get more insight into what is happening?


On a side note, the "Feature skipped." messages seem to be printed directly to 
the
screen and not to stderr or stdout, as I cannot redirect them.

Original comment by peppe...@gmail.com on 27 Feb 2009 at 8:01

[GoogleCodeExporter]

I noticed my count files I had also failed to make 6-gram, so I rebuilt these, 
but
the error messages persisted. I removed the error message from the code, 
rebuilt, and
restarted the 4-model build, and it completed with decent success after about 17
minutes (it seemed the error message was slowing things down a lot). I removed 
the
log^2 from the features however, because of the error messages, and it had not
completed after about 80 minutes. Still unsure as to why the error messages 
occur,
however.

Original comment by peppe...@gmail.com on 27 Feb 2009 at 10:13

[GoogleCodeExporter]

One more comment. Even though I made it go through ignoring the errors, it 
seems they
still represent some problem, as my perplexities are significantly higher than 
LI for
CM (CM-52 vs LI-46, CM-37 vs LI-33), and marginally lower to marginally higher 
than
LI for GLI (GLI-48.5 vs LI-46.,3, GLI-32.3 vs LI 33.3) with log:sumhist.

Original comment by peppe...@gmail.com on 27 Feb 2009 at 11:00