BART Support

[anshoomehra]

@jalammar et al,

Can't thank you enough for your thoughtful approach with "Illustrated Transformers" & for further uncovering Transformers (visually) with the help of "Ecco". These articles not just helping folks fundamentally understand things better but are acting as a stimulator for responsible future research. So thank you for the great work!! [ I apologize for the distraction, couldn't resist my gratitude while I had this opportunity to write to you! ]

I am primarily working on BART, T5, PEGASUS base & fine-tuned variants for my research on summarization & extreme summarization objectives for my company. I understand that ecco support generative models, perhaps the support for mentioned is not there yet.

1. Not sure, if these can be easily accommodated & your team is already in process of adding them?
2. If not, I am sure you must be overwhelmed with similar asks, I (& am sure rest) would love to participate. I have not gone through the code yet, but if there is some jump start documentation for the code, specifically on how to refactor the code for a different type of models, perhaps we can help contribute enriching this beautiful library.
3. While the above is looked into, would you have a suggestion of any other library which can support similar outputs for BART and Pegasus for now ??

Best,
Anshoo

[jalammar]

Hi Anshoo,

Thank you so much for your kind words!

1. We are currently working on BERT support for activation exploration. We don't have immediate plans for supporting other models but we're listening to the community.
2. We are working on more extensive documentation that should make it easier for people to adapt Ecco for their needs and use-cases. Thank you for the initiative.
3. Your best bet is Captum - https://captum.ai/tutorials/Bert_SQUAD_Interpret

[anshoomehra]

Hi Anshoo,

Thank you so much for your kind words!

1. We are currently working on BERT support for activation exploration. We don't have immediate plans for supporting other models but we're listening to the community.
2. We are working on more extensive documentation that should make it easier for people to adapt Ecco for their needs and use-cases. Thank you for the initiative.
3. Your best bet is Captum - https://captum.ai/tutorials/Bert_SQUAD_Interpret

@jalammar Appreciate your time responding to this inquiry & the reference to Captum.

[yegorkryukov]

@jalammar Looks like 'facebook/bart-large-mnli' is now supported?

When I instantiate it I get the following error:

lm = ecco.from_pretrained('facebook/bart-large-mnli')

AttributeError                            Traceback (most recent call last)

<ipython-input-5-a9021088d96a> in <module>()
      1 import ecco
----> 2 lm = ecco.from_pretrained('facebook/bart-large-mnli')

2 frames

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in __getattr__(self, name)
   1176                 return modules[name]
   1177         raise AttributeError("'{}' object has no attribute '{}'".format(
-> 1178             type(self).__name__, name))
   1179 
   1180     def __setattr__(self, name: str, value: Union[Tensor, 'Module']) -> None:

AttributeError: 'BartForConditionalGeneration' object has no attribute 'shared'

Any suggestions on how I can fix this?

Thanks in advance!

[jalammar]

Unfortunately BART support wasn't completely there yet. I just made a couple of updates, could you try it now?

Install the latest version of the repo:
!pip install git+https://github.com/jalammar/ecco.git

[yegorkryukov]

Thank you Jay. The model now loads.

I'm trying to understand how to use it with the MNLI model. That model accepts a premise and a label. Where the premise is a piece of text ('Computer shut down unexpectedly') and a label ('politics'). So, the model spits out its prediction whether the label is correct for the premise.

I can't figure out how to pass those two elements (premise and label) to any functions in the ecco lm. Any advice?

[jalammar]

My guess would be that that they put them in the same string separated by the [sep] token. The paper or model docs should explain that, I think.

[Lim-Sung-Jun]

Hello, @jalammar

Is there example of using a BERT?
I've seen a lot examples of GPT..

I want to use saliency on BERT's head, not INPUT TOKEN.

[jalammar]

@Lim-Sung-Jun There are examples of using BERT for neuron activation, but not for primary feature attribution/saliency. That hasn't been built out yet. I have create issue #64 to track the feature request and invite people to support it.

[Lim-Sung-Jun]

@Lim-Sung-Jun There are examples of using BERT for neuron activation, but not for primary feature attribution/saliency. That hasn't been built out yet. I have create issue #64 to track the feature request and invite people to support it.

I really appreciate your support!