Skip to content

allow attention processors to have different signatures #6915

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Feb 10, 2024
Merged

allow attention processors to have different signatures #6915

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions src/diffusers/models/attention_processor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import inspect
from importlib import import_module
from typing import Callable, Optional, Union

Expand Down Expand Up @@ -509,6 +510,15 @@ def forward(
# The `Attention` class can call different attention processors / attention functions
# here we simply pass along all tensors to the selected processor class
# For standard processors that are defined here, `**cross_attention_kwargs` is empty

attn_parameters = set(inspect.signature(self.processor.__call__).parameters.keys())
unused_kwargs = [k for k, _ in cross_attention_kwargs.items() if k not in attn_parameters]
if len(unused_kwargs) > 0:
logger.warning(
f"cross_attention_kwargs {unused_kwargs} are not expected by {self.processor.__class__.__name__} and will be ignored."
)
cross_attention_kwargs = {k: w for k, w in cross_attention_kwargs.items() if k in attn_parameters}
Copy link
Member

@sayakpaul sayakpaul Feb 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could I get an explanation on how this ensures the scale parameter from cross_attention_kwargs gets passed appropriately?

Copy link
Collaborator Author

@yiyixuxu yiyixuxu Feb 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It won't get passed if the attention processor does not accept a scale parameter - it will throw a warning instead.

For example, IPAdapterAttnProcessor has a scale parameter in its signature, but it does not use it. We only added it so that we can use it together with the default attention processors

diffusers/src/diffusers/models/attention_processor.py

Line 2133 in f07899a

scale=1.0,

With this, we don't need to do that anymore. - Attention Processors will have fixed signatures that only include parameters that they actually use

Copy link
Collaborator Author

@yiyixuxu yiyixuxu Feb 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

another example is in this PR https://github.com/huggingface/diffusers/pull/6847/files#diff-96aeb208ac8e69bdb368e0a481a0cd50173b3f4bf19e5da500c251614820c8cdR1201

it introduced a new parameter for IPAdapterAttnProcessor ip_adapter_mask - and they had to add same parameter to AttenProcessor.call in order for these two attention processors to work together. We don't need to do that too

Copy link
Member

@sayakpaul sayakpaul Feb 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for explaining. The PR is now a go.


return self.processor(
self,
hidden_states,
Expand Down