[🐛 Bug Report]: IndexOutOfBoundsException while training the model

[borisrakovan]

Describe the bug

I am trying to train a simple model using the PyNeraLogic framework on the CLUTRR dataset.

However, after creating my Dataset and setting up the Template, I am getting the following error before the first training epoch starts:

/usr/local/lib/python3.7/dist-packages/_jpype.cpython-37m-x86_64-linux-gnu.so in cz.cvut.fel.ida.neural.networks.computation.training.strategies.PythonTrainingStrategy.learnSamples()

/usr/local/lib/python3.7/dist-packages/_jpype.cpython-37m-x86_64-linux-gnu.so in cz.cvut.fel.ida.neural.networks.computation.training.strategies.trainers.SequentialTrainer$SequentialListTrainer.learnEpoch()

/usr/local/lib/python3.7/dist-packages/_jpype.cpython-37m-x86_64-linux-gnu.so in cz.cvut.fel.ida.neural.networks.computation.training.strategies.trainers.Trainer.learnFromSample()

/usr/local/lib/python3.7/dist-packages/_jpype.cpython-37m-x86_64-linux-gnu.so in cz.cvut.fel.ida.neural.networks.computation.training.strategies.trainers.Trainer.backpropSample()

/usr/local/lib/python3.7/dist-packages/_jpype.cpython-37m-x86_64-linux-gnu.so in cz.cvut.fel.ida.neural.networks.computation.iteration.actions.Backpropagation.backpropagate()

/usr/local/lib/python3.7/dist-packages/_jpype.cpython-37m-x86_64-linux-gnu.so in cz.cvut.fel.ida.neural.networks.computation.iteration.modes.Topologic$TDownVisitor.topdown()

/usr/local/lib/python3.7/dist-packages/_jpype.cpython-37m-x86_64-linux-gnu.so in java.util.ArrayList.get()

/usr/local/lib/python3.7/dist-packages/_jpype.cpython-37m-x86_64-linux-gnu.so in java.util.Objects.checkIndex()

/usr/local/lib/python3.7/dist-packages/_jpype.cpython-37m-x86_64-linux-gnu.so in jdk.internal.util.Preconditions.checkIndex()

/usr/local/lib/python3.7/dist-packages/_jpype.cpython-37m-x86_64-linux-gnu.so in jdk.internal.util.Preconditions.outOfBoundsCheckIndex()

/usr/local/lib/python3.7/dist-packages/_jpype.cpython-37m-x86_64-linux-gnu.so in jdk.internal.util.Preconditions.outOfBounds()

Exception: Java Exception

The above exception was the direct cause of the following exception:

java.lang.IndexOutOfBoundsException       Traceback (most recent call last)
[<ipython-input-96-4b77102c2769>](https://localhost:8080/#) in <module>()
      5 average_losses = []
      6 
----> 7 for i, current_total_loss, number_of_samples in enumerate(evaluator.train(dataset)):
      8     print(f"epoch: {i}")
      9     clear_output(wait=True)

[/usr/local/lib/python3.7/dist-packages/neuralogic/nn/evaluators/java.py](https://localhost:8080/#) in _train()
     42         def _train():
     43             for _ in range(epochs):
---> 44                 results, total_len = self.neuralogic_model(None, True)
     45                 yield sum(result[2] for result in results), total_len
     46             if dataset is not None:

[/usr/local/lib/python3.7/dist-packages/neuralogic/nn/java.py](https://localhost:8080/#) in __call__(self, samples, train, auto_backprop, epochs)
     80 
     81         if samples is None:
---> 82             results = self.strategy.learnSamples(epochs)
     83             deserialized_results = json.loads(str(results))
     84 

java.lang.IndexOutOfBoundsException: java.lang.IndexOutOfBoundsException: Index -1 out of bounds for length 9

Steps to reproduce the behavior

My Template is based on this example.

One sample graph in the CLUTRR dataset looks as follows

story: [('Donald', 'father', 'Michael'), ('Michael', 'sister', 'Dorothy')]	
target: ('Donald', 'aunt', 'Dorothy')

The input triplets and the target pair (Donald, Dorothy) are given and the task is to predict their relationship (aunt), it is thus an R-way classification task where R is the total number of relations (such as aunt, father, ...).

Below you can see the dataset creation process

for i in range(len(data.train)):
    sample = data.train[i]

    query = Relation.predict(*sample.target)
    
    persons = set(p for p, _, _ in sample.story)
    persons.update(p for _, _, p in sample.story)

    example = list(itertools.chain(
        (Relation.edge(u, r, v) for u, r, v in sample.story),
        (Relation.rel(r) for _, r, _ in sample.story + [sample.target]),
        (Relation.person(p) for p in persons)
    ))

    dataset.add_example(example)
    dataset.add_query(query)

This results in a dataset that looks like this:

Query:
predict(Donald, aunt, Dorothy).
Example:
['edge(Donald, father, Michael)', 'edge(Michael, sister, Dorothy)', 'rel(father)', 'rel(sister)', 'rel(aunt)', 'person(Dorothy)', 'person(Donald)', 'person(Michael)']

The template definition is equivalent to the definition of the example template mentioned above, but it is rewritten to the PyNeuraLogic syntax. It looks as follows:

template = Template()

# {3} @embed_nation(X) :- nation(X).
# {3} @embed_rel(X) :- rel(X).
template.add_rules([
    (R.person_embed(V.A)[3,] <= R.person(V.A)),
    (R.rel_embed(V.B)[3,] <= R.rel(V.B))
])

# embed_nat1(X) :- {3,3} embed_rel(R), {3,3} embed_nation(Y), {3} r(X,R,Y).
template.add_rule(
    R.person_embed1(V.X) <= (R.rel_embed(V.R)[3, 3], R.person_embed(V.Y)[3, 3], R.edge(V.X, V.R, V.Y)[3,])
)

# embed_nat2(Y) :- {3,3} embed_nation(X), {3,3} embed_rel(R), {3} r(X,R,Y).
template.add_rule(
    R.person_embed2(V.Y) <= (R.person_embed(V.X)[3, 3], R.rel_embed(V.R)[3, 3], R.edge(V.X, V.R, V.Y)[3,])
)

# {1,3} predict(X,R,Y) :- {3,3} embed_nat1(X), {3,3} embed_nat2(Y), {3,3} embed_rel(R).
template.add_rule(
    R.predict(V.X, V.R, V.Y)[1, 3] <= (R.person_embed1(V.X)[3, 3], R.person_embed2(V.Y)[3, 3], R.rel_embed(V.R)[3, 3])
)

The evaluator definition and part of training loop can be seen here. The last line triggers the exception.

settings = Settings(optimizer=Optimizer.ADAM, epochs=120, learning_rate=0.001, error_function=ErrorFunction.SQUARED_DIFF)
evaluator = get_evaluator(template, Backend.JAVA, settings)

for i, current_total_loss, number_of_samples in enumerate(evaluator.train(dataset)):
    ...

Expected behavior

calling evaluator.train(dataset) should yield (current_total_loss, number_of_samples) correctly instead of raising an exception

Environment

Google Colab (Python 3.6)

Additional context

Also, are there any other fundamental issues with the way I defined the dataset and architecture for this particular task? I am still trying to wrap my head around the way this framework works and how to rewrite PyG code to neuralogic language.

Thank you!

[LukasZahradnik]

Hi @borisrakovan

Thank you for opening the issue. I can't reproduce the same error as you reported, but your dataset (both example and query) has an issue that might be the cause of your problem.

In (Py)NeuraLogic, you can have two types of terms - logical variables (represented by strings) and constants (represented by numeric values or strings). I assume that "Dorothy", "Michael", and "Donald" are supposed to be specific entities, which should be represented as constants. Constants represented as strings have to start (by convention) with lower case letters ("dorothy", "donald" etc.)

Your template seems to be okay. The only thing is that those two rules in the first comment are not equivalent to the rules you wrote in PyNeuraLogic (your version is absolutely valid, just not equivalent).

# {3} @embed_nation(X) :- nation(X).
# {3} @embed_rel(X) :- rel(X).

The equivalent to that would be:

template.add_rules([
    (R.special.embed_person(V.A)[3,] <= R.person(V.A)),
    (R.special.embed_rel(V.B)[3,] <= R.rel(V.B))
])

The @embed_x special predicate is disabled in the latest NeuraLogic (I believe) due to some issues. So (actually working) equivalent would be:

template.add_rules([
    *[(R.person_embed(p)[3,] <= R.person(p)) for p in ("dorothy", "donald", "michael")],
    *[(R.rel_embed(r)[3,] <= R.rel(r)) for r in ("father", "sister", "aunt")]
])

I listed only entities (people and relations) from your provided example, but the idea is to list all entities and embed them into unique (not shared) vectors [3,] (just like in embeddings.txt here)

R.person_embed(V.A)[3,] <= R.person(V.A)

This (your) rule would embed all possible entities into one shared vector.

[borisrakovan]

Hi @LukasZahradnik,

thank you very much for your prompt response.

I fixed both the issues that you pointed out and I was able to proceed with training the model. However, now I encountered a different issue. From the very first epoch, the loss seems to be already near 0, and it stays pretty much the same during the rest of epochs.

These are some debug logs to illustrate:

epoch: 0
loss=8.215885599999999e-09
epoch: 1
loss=8.058118199999975e-09
epoch: 2
loss=7.906300199999991e-09
epoch: 3
loss=7.760098199999983e-09
epoch: 4
loss=7.619198799999999e-09

I tried to examine different possibilities to find out what's wrong but I couldn't find much. The only thing I noticed is that the
dataset.number_of_classes value is equal to 1. Also, the when testing the model using for y, y_hat in evaluator.test(dataset[:100]), the output seems as if there was only 1 class that is always predicted:

Expected: 1.0, Predicted: 1 (0.9999991647549823)
Expected: 1.0, Predicted: 1 (0.9999990914407968)
Expected: 1.0, Predicted: 1 (0.999999167178837)
Expected: 1.0, Predicted: 1 (0.9999990778106539)
Expected: 1.0, Predicted: 1 (0.9999991579194215)
Expected: 1.0, Predicted: 1 (0.9999992025083247)
Expected: 1.0, Predicted: 1 (0.9999990467822975)

It looks like there is something wrong with the queries that I pass to the dataset. I don't really understand how the NeuraLogic interprets the queries and how it translates them to actual classes.

Do you know what could be the problem here?

[GustikS]

Do you have also negative examples in your queries? Like in the nations dataset, where it's 50:50.
If you don't have those you typically sample them by corrupting the positive ones.
There are some methods for that in the backend: https://github.com/GustikS/NeuraLogic/blob/ac33e94558ba1a17c44eb5403024e9eef4f05d9e/Learning/src/main/java/cz/cvut/fel/ida/learning/results/metrics/HITS.java
but with python you are probably better off just generating them yourself at the moment.

[LukasZahradnik]

Hi @borisrakovan

firstly, dataset.number_of_classes value is relevant only for datasets using the Data class - datasets where it is needed to translate tensor (e.g., torch) representation into logic representation. It is an argument used to specify how long the one-hot encoded vector (representing the output, i.e., the y value) should be.

For example:

Dataset(data[Data(x=..., edge_index=..., y=2)], one_hot_encode_labels=True, number_of_classes=3)
# query would be [0, 0, 1] predict.

Dataset(data[Data(x=..., edge_index=..., y=2)], one_hot_encode_labels=True, number_of_classes=4)
# query would be [0, 0, 1, 0] predict.

Regarding the issue you have, it looks like you are providing only positive examples (as @GustikS stated above). When your queries have no labels (y values), PyNeuraLogic sets the label to 1.0 (so predict(donald, aunt, dorothy) will be modified to 1.0 predict(donald, aunt, dorothy).

In your case, you might want to add some negative examples with queries such as R.predict("donald", "sister", "dorothy")[0] (the y is 0 here).

[borisrakovan]

@GustikS @LukasZahradnik thank you for your responses, I generated some negative example-query pairs and now the training works as expected.

However, I realized that with the current setting, the model performs a binary classification task, which is not quite the task that I intended to perform initially with CLUTRR. What I wanted to achieve instead is a multi class classification of the query entity pair based on the input graph. Now I have to provide the model with a query (entityA, relation, entityB) and the model outputs 0/1 based on whether it thinks that the relation holds between the two entities. Instead, I would like to only input (entityA, entityB) and have the model predict the correct relation out of all possible (20) relations.

What changes would I need to make to the model definition to be able to perform this multiclass classification task? Is there maybe a better example of multiclass classification performed in Neuralogic, since the one I used as a template no longer seems to be appropriate for my use case?

My first idea was to keep the examples the same and only alter the queries in the following way

R.predict(entityA, entityB)[0,0,1,0,...]

Where the parameter is a one-hot encoded vector representing the correct relation. Is this a valid line of thought? If so, how would I need to alter the template?

Thank you.

[GustikS]

Hi, ok I didn't realize they treat it as a multiclass classification and not knowledge-base completion.
What you propose makes sense to me and should work, I guess, i.e. encoding the relation as one-hot instead of a logical term (just do it everywhere accordingly, including the examples) and having a multiclass label (that should put softmax on top under the hood). That will make it close to a classic implementation (RGCNN-like) in standard frameworks.

However, when you move it to vectors/tensors like that you lose the relational expressiveness, i.e. ability to work with the term/relations in more sophisticated ways, e.g. changing the propagation scheme based on the relation. For that you'd want to keep the logical encoding and just "brute-force" enlist all the 20 queries (19 negative and 1 positive) with each example graph.
It's not exactly the same since there isn't the softmax on top, so it might work a bit worse (I remember I was planning to implement something like that in the backend, so I might brush up on that).

Eventually, you might play with a combination of the two approaches, i.e. supply both the one-hot encoding and the logical relation term. That will require some thinking/experimentation when modifying the template, but some interesting modelling constructs might come out of that...

[GustikS]

Since this is not a bug but rather a discussion, let us move it there...