Using NL for multiclass classification task

[borisrakovan]

Hello,

In the past days I was trying to figure out how to use the NeuraLogic framework to imitate the computation performed by Relational graph convolutional networks (RGCN) introduced here. Unfortunately, I wasn't able to come up with any working solution. I just can't seem to understand how to translate such computations into the logic-based language.

I have my examples and queries set up in a way that is very similar to the one already discussed here . I would like to perform a multiclass classification task (with 22 classes) using the following propagation rule:

In RGCNs, the transformation applied to the neighboring nodes depends on the particular type of the relation (category) between the two nodes, and I can't figure out how to encode this using the NL syntax. I managed to come up with an initialization of the embeddings:

template2.add_rules([
    *[(R.person_embed(p)[3,] <= R.person(p)) for p in all_persons],
    *[(R.rel_embed(r)[3, 3] <= R.rel(r)) for r in all_relations]
])

where I initialize a 3x3 weight matrix for each of the relations, but I don't know whether this is the correct line of thinking. Alternatively, what would be the simplest way to set up a template for multiclass classification on this dataset - one that I could use as a starting point?

Any help would be much appreciated! Thank you.

[LukasZahradnik]

Hi, I have recently written a short tutorial on rules mapping that might help you understand NL syntax more.

One layer of RGCN could be rewritten as (I added comments on top of the rule to make mapping more clear; also, the multiplication is the other way around in the comment - it is actually W * h):

t = Template()
metadata = [Aggregation.SUM, Activation.IDENTITY]

for relation in all_relations:
    #       h_i1           h_j0  *  W_r +  h_i0  *  W_0         Ensuring j \in N^r_i
    t += (R.h_1(V.I) <= (R.h_0(V.J)[1,], R.h_0(V.I)["W_0": 1,], R.hidden.edge(V.I, relation, V.J))) | metadata
    t += R.h_1 / 1 | [Activation.SIGMOID]

For additional layers, you would change only indices, that is:

    #                                               W_0 in the 1st layer 
    t += (R.h_2(V.I) <= (R.h_1(V.J)[1,], R.h_1(V.I)["W_1": 1,], R.hidden.edge(V.I, relation, V.J))) | metadata
    t += R.h_2 / 1 | [Activation.SIGMOID]

You will have one rule for each relation, R.h_1, which computes the inner sum (the sum is defined in the attached metadata). Getting all j's from N_ir is done automatically in the background (enforcing j's to be from N_ir is done with R.edge, without it, all j's would be taken into account).

The outer sum over relations is done in the background as well - rules with the same (grounded) head are summed when queried.

The only part I omitted is the normalization constant - it's a bit tricky, and I'm not sure how you want to define it.

You could, for example, add it into the dataset examples as:

    R.c("entity_1", "relation_1")[0.5]  # Some value c_{i,r}

And then change the template a little bit (the new (first) rule computes c_i,r * Wr * h0j)

for relation in all_relations:
    #                                         C_i,r        * W_r * h_j0                 
    t += (R.c_0(V.I, relation, V.J) <= (R.c(V.I, relation), R.h_0(V.J)[1,])) | Metadata(activation="product-identity")
    # "product-" enforces multiplicating instead of summing body relations
    # That's why it is (C_i,r * W_r * h_j0) and not (C_i,r + W_r * h_j0)
    # the 'r' is equal to 'relation'

    t += R.c_0 / 3 | [Activation.IDENTITY]

    #      h_i1         (C_i,r * W_r * h_j0)       +  h_i0  *  W_0         Ensuring j \in N^r_i
    t += (R.h_1(V.I) <= (R.c_0(V.I, relation, V.J), R.h_0(V.I)["W_0": 1,], R.hidden.edge(V.I, relation, V.J))) | metadata
    t += R.h_1 / 1 | [Activation.SIGMOID]

I haven't really tested it, but it should correspond to your RGCN definition. When it comes to the multiclass classification, you should either go with a one-hot encoded vector or make a query for each possible combination of entities and relations (where you will learn query for the correct class to be 1 and others to be 0).

// EDIT: I tested the template, visualized it, and it seems to generate the correct computation graph.

[GustikS]

Hi, you can still attach learnable weights to the relations (edge) and rule heads (h1 and h2) and replace the identities with some non-linearity (e.g. tanh, since ReLU could still be causing linear behavior in these small networks). Or just simply remove all the metadata/activations, really.

But all that is really just for further brute-force capacity increase. I'd strongly suggest to avoid these black-box trials and carefully inspect the model structure instead (as suggested above). Particularly, inspect whether the information necessary to distinguish the samples does not get lost somewhere during the inference (causing the random behavior). Personally, I'd start with just taking 2 samples you know exactly how to distinguish (e.g. manually crafted), inspecting the model inference (via visualization or debugger), and try to e.g. manually set-up the weights so that you confirm the model has the capacity to correctly classify these (and see why!). Then try to learn these weights automatically (and, potentially, see why not). Then add some more, principally different sample (e.g. more relations, further apart, etc.), and repeat. Once this gets smooth across different sample types, you can move to the black-box tuning mode on the full dataset.

Again, if you don't feel comfortable debugging the model like that, you can use the Java backend just as a grounder ("neural model creator"), export to Dynet/Torch, and continue debugging from there (or even manually create the models from scratch there, once you understand the model principle/structure). This could be also good to confirm the alignment, and potential discrepancies, from your reference RGCN implementation.

For instance, one thing I notice that might be different from the RGCN formula is that the W_0 * h_i is outside of the sums (i.e. you're not adding that for each relation and each neighbor), so you might want to move that into a separate rule (no need to name the weight):

R.h_1(V.I) <= R.h_0(V.I)[10,10]

Regarding the logs - you can filter it via standard logging levels. But it seems that your logging level is set to INFO only. @LukasZahradnik did you limit that? If so, can you give option to turn that to at least FINE or FINER - that would include more useful info about the grounding here (would be good to see the Herbrand sizes etc.). Also, on my system it shows time deltas in milliseconds (that would be more helpful) - @LukasZahradnik, did you turn that off somehow? :) Perhaps it's the Colab formatter - I'd suggest running locally for this.

From the warning, it seems that the GC is throttling the process (this means that you've got eventually completely blocked by GC on 2 threads), which might explain why the grounding gets stuck around the 2000 samples (as I speculated above, or is it slow from the start?). To avoid this you can increase the Java virtual memory by passing e.g. -Xmx4g to the JVM (default is 512m I think). But if the grounding is generally slow from the start, there will be some problem with the model instead (which will be seen from the detailed logs, too).

I wish you strong nerves :)

[LukasZahradnik]

Hi,

@GustikS I'm not limiting anything. I'm only setting the Java's stdout and stderr, which I assumed would be enough. It seems like I will have to work directly with loggers instead, which would also resolve (re)setting stdout/stderr problem @borisrakovan mentioned. There might be some issues because each class has its own logger, but we will see...

For instance, one thing I notice that might be different from the RGCN formula is that the W_0 * h_i is outside of the sums

This is kinda my bad. I overlooked it while rewriting it into PyNeuraLogic. Regarding the GC issue, maybe it might make sense to trigger GC manually from time to time in the backend/frontend?

@borisrakovan if you choose to go with DyNet/Torch, I would recommend you to go with Torch, as PyNeuraLogic has implemented better support for it (correct broadcasting etc.).
Also, increasing the memory is not possible in the latest release. I have it implemented and will release it soon with some other changes.

[LukasZahradnik]

Hello, I fixed the logging in the latest version (0.1.4). The usage is following:

import sys

from neuralogic.logging import add_handlers, clear_handlers


add_handlers(sys.stdout)  # by default FINER and COLOR Formatter

clear_handlers()  # Clear ALL handlers

[borisrakovan]

Hi @GustikS @LukasZahradnik ,
I am trying to use the Torch backend as suggested, but it gives me NotImplementedError. From the code it looks like the crossentropy loss is indeed not implemented:

if error_function not in TorchEvaluator.error_functions:
    raise NotImplementedError

Also, as a side note, the grounding process is faster on my local machine then it is in Colab - it only takes about ~2 minutes for 1600 samples, which is still not ideal, but it's much better than ~10 minutes in Colab.

[LukasZahradnik]

Hi @borisrakovan

You can add the Crossentropy error function yourself by adding it into the dictionary here (just import the class and add it into the dictionary), or you can define your own training loop. I will try to add it to one of the next releases.

I'm checking out your problem, and it is indeed slow. I will debug the model and let you know later on.