DLRM Tutorial

[123nishant]

🐛 Bug

On executing the code in DLRM tutorial I try to clone the official repository of DLRM .
Yet i am facing errors considering 'apply_emb' and 'IntegratedGradients' .
You have mentioned versions of dlrm_s_pytorch .

I am a bit new and I have searched through a lot of sites.
Can you please help me out how to use the different versions ?

I have used the version of DLRM in the master branch but I am not able to proceed.
Need help here.

Error Message :

1. embeddings = dlrm.apply_emb(lS_o_test, lS_i_test, dlrm.emb_l)

---

TypeError Traceback (most recent call last)

in ()
----> 1 embeddings = dlrm.apply_emb(lS_o_test, lS_i_test, dlrm.emb_l)

TypeError: apply_emb() missing 1 required positional argument: 'v_W_l'

2. ig = IntegratedGradients(sequential_forward)

---

NameError Traceback (most recent call last)

in ()
----> 1 ig = IntegratedGradients(sequential_forward)

NameError: name 'IntegratedGradients' is not defined

Expected behavior

Environment

• Captum / PyTorch Version (e.g., 1.0 / 0.4.0):
• Colab (with GPU)
• How you installed Captum / PyTorch (conda, pip, source):
• Build command you used (if compiling from source):
• Python version:
• CUDA/cuDNN version:
• GPU models and configuration:
• Any other relevant information:

Additional context

Need Help .
Please let me know how can I proceed with this.

[bilalsal]

Hi @123nishant ,

indeed the relevant DLMR API has changed in facebookresearch/dlrm#142 requiring v_W_l as an additional argument to apply_emb to support weighted pooling.

As a workaround, you can provide a list of None:

v_W_l =   [None] * len(lS_i_test)
embeddings = dlrm.apply_emb(lS_o_test, lS_i_test, dlrm.emb_l, v_W_l)

Alternatively, you can use dlrm.create_emb(...) by passing the [needed parameters](https://github.com/facebookresearch/dlrm/blob/master/dlrm_s_pytorch.py#L236).

Regarding the issue with IntegratedGradients, are you sure you ran this line in the 2nd cell?

from captum.attr import IntegratedGradients, LayerConductance, NeuronConductance

Hope this helps

[123nishant]

HI @bilalsal , thank you for the details .

I'm checking it out.
Is there any place where i can check out the details regarding input of some functions ?

It's just that I searched a lot but couldn't find a documentation regarding this algorithm (DLRM) , explaining the inputs for the functions .

Few of these variables, were a bit difficult to decode. One of them being v_w_l.

Is there something that I can refer to for this ?

[123nishant]

HI @bilalsal , I'll trouble you a bit more .

Below is my code, which is triggering a "CUDA error: device-side assert triggered".

For the "CUDA error: device-side assert triggered" , I tried details mentioned here :

1. https://stackoverflow.com/questions/55780923/what-does-runtimeerror-cuda-error-device-side-assert-triggered-in-pytorch-me
2. https://stackoverflow.com/questions/14038589/what-is-the-canonical-way-to-check-for-errors-using-the-cuda-runtime-api

But cudamemcheck does not give me errors.
I think so it should be run parallely, and I'm searching for a way how to do so in Google Colab Notebooks.

Can you help me out here ?

[bilalsal]

Hi @123nishant ,

what about running the script completely on CPU?
Change cell #3 to: device = torch.device("cpu")

Hope this helps

[123nishant]

Hi @bilalsal ,

I tried doing that .

It shows an error much before this cell, just after when we download the model using wget.

Looks like this notebook was made to run on GPU

[bilalsal]

Hi @123nishant

What is the error you got when using CPU?
That would be easier to debug, I hope.

[123nishant]

HI @bilalsal ,

It was a mistake from my end.
It works now using the CPU.

However, in later scenarios if I need to use CUDA , how can we address this scenario ?

[123nishant]

Also I guess it's a basic question , but I'm getting this error :

I'm linking my Colab notebook here. I hope it helps out to understand my predicament.
https://colab.research.google.com/drive/1RqJ3m2FOURafZDGZcp5vw484-NCMYy_O?usp=sharing

[bilalsal]

Hi @123nishant ,

in your notebook, the first attribution with lS_o_test works fine as it seems to have healthy starts:

print(lS_o_test)
tensor([[ 0,  1,  2,  ..., 97, 98, 99],
        [ 0,  1,  2,  ..., 97, 98, 99],
        [ 0,  1,  2,  ..., 97, 98, 99],
        ...,
        [ 0,  1,  2,  ..., 97, 98, 99],
        [ 0,  1,  2,  ..., 97, 98, 99],
        [ 0,  1,  2,  ..., 97, 98, 99]])

whereas the attribution with lS_o_test_above_0999 fails as the first offset values are 28 not 0s:

print(lS_o_test_above_0999)

tensor([[28,  0, 40,  ..., 24, 42, 53],
        [28,  0, 40,  ..., 24, 42, 53],
        [28,  0, 40,  ..., 24, 42, 53],
        ...,
        [28,  0, 40,  ..., 24, 42, 53],
        [28,  0, 40,  ..., 24, 42, 53],
        [28,  0, 40,  ..., 24, 42, 53]])

Is it possible that the change in the DLRM API led to this issue?
I am not quite familiar with DLRM.
Maybe try to run the notebook with a former version of DLRM? The key is to find out why your offset values are not starting with 0s.
With a bit of debugging you should find a good solution, esp. now that Captum works on lS_o_test.

[123nishant]

HI @bilalsal , I checked and the offset values in "lS_o_test_above_0999" are inherently generated using the data present in the "Captum Tutorial Directory" : [ https://github.com/pytorch/captum/tree/master/tutorials/data/dlrm ]

A thought , this issue is generated not due to DLRM api but due to this data.

Can it be generated due to changes in DLRM api ?
Maybe .

But which version does it works with ?

[bilalsal]

Hi @123nishant , good point! The code and the data should work against this version of DLMR:
https://github.com/facebookresearch/dlrm/tree/74c8b8dfe93d7beeafc07951ebb65b8fd19e8fb3

[123nishant]

HI @bilalsal ,

Thanks for the link . It helped me a lot.
I have used the repo that you shared me.

However, the error still persists.

I believe the error is from the data stored in "https://github.com/pytorch/captum/blob/master/tutorials/data/dlrm/X_S_T_test_above_0999"

I request you to check that out once.
It needs a modification.

[NarineK]

Hi @123nishant,

It looks like the generated datasets, that we refer to in the tutorial, are outdated. I updated both datasets. Do you mind using the following versions of the datasets and letting us know if you are still seeing the issue?

https://pytorch-tutorial-assets.s3.amazonaws.com/captum/dlrm/X_S_T_test
https://pytorch-tutorial-assets.s3.amazonaws.com/captum/dlrm/X_S_T_test_above_0999

Let us know if you are still seeing the issue both in GPU and CPU environments.

Please, pass both dlrm.emb_l and dlrm.v_W_l when generating the embeddings.

Thank you!

[123nishant]

@NarineK , Thanks for the update .

I will check it out.