JVM Crash When ID Doesn't Exist

[iprovalo]

    @Test
    public void testJvmCrashLoad() {
        Assert.assertEquals(null, sp.idToPiece(38067));
    }

This test will crash jvm.

[levyfan]

This is caused by the sentence piece native code const std::string &piece = spp->IdToPiece(id);. Please check their issues track.

[levyfan]

I think there is a clear warning that id should be valid.

  // Returns the string representation of vocab with `id`.
  // id must be 0 <= id < GetPieceSize().
  virtual const std::string &IdToPiece(int id) const {
    return model_proto_->pieces(id).piece();
  }

https://github.com/google/sentencepiece/blob/02555b8f1994cd05e4c22e3c4fe55220134e518a/src/model_interface.h#L137

[levyfan]

I add a size check to avoid JVM crash here #29, now it will throw an exception that you can catch.

[iprovalo]

thank you very much, @levyfan !