Skip to content
This repository was archived by the owner on Jul 7, 2023. It is now read-only.
This repository was archived by the owner on Jul 7, 2023. It is now read-only.

dot_product_attention_relative query/value compatibility #1652

@davek44

Description

@davek44

The common_attention method dot_product_attention_relative requires that the query and value have compatible depth. See

q.get_shape().assert_is_compatible_with(v.get_shape())

While that's certainly true for the query and key, the value is handled separately and doesn't interact. I've commented out the compatibility assertion and verified that the method runs well and models using it train well. Am I missing something here or can this line be removed?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions