Getting an error when running the code SelfAttentionv1(d_in,d_out)

[AIstatistican]

hi, when running the code
SelfAttentionv1(d_in,d_out)
print(sa_v1(input))

ıt gets an error :"expected m1 and m2 to have the same dtype, but got: long long != float"

what is the solution?

Thanks.

[rasbt]

Hi there,

do you mean SelfAttention_v1 in chapter 3?

Let's find out what's going on your case.
Could you show the full error message to track down in which particular line the error occurs?

[AIstatistican]

Thanks for your replies. I am waiting the chapter 7 with great excitement. When it gave me that error, I have converted the inputs that was established from;
dataloader = create_dataloader_v1(raw_text, batch_size=256, max_length=256, stride=4, shuffle=False)
data_iter = iter(dataloader)
inputs, targets = next(data_iter)
print("Inputs:\n", inputs)

by using;
inputs=inputs.float()

then it did not give any error and printed the results.

[AIstatistican]

So, could I understand that we created a GPT model in chapter 4 and the text file you provided as "verdict.txt" is only used for demonstration not for creating a GPT model.

[rasbt]

Thanks for the info. May I ask why the inputs were int64? Ideally, they should have been float32 via the following:

Could it perhaps been a typo? (Sorry for the many questions, just want to make sure that there are no weird behaviors on certain operating systems or PyTorch versions).

o, could I understand that we created a GPT model in chapter 4 and the text file you provided as "verdict.txt" is only used for demonstration not for creating a GPT model.

As far as I recall, I didn't use the "the-verdict.txt" file in chapter 4, only in chapters 2 and 5. Chapter 4 is setting up the architecture, and chapter 5 is training it.

[AIstatistican]

Yes, that is right. But, I used my own txt file for demonstrating in every steps.

when i came to that code;

dataloader = create_dataloader_v1(raw_text, batch_size=256, max_length=256, stride=4, shuffle=False)

data_iter = iter(dataloader)
inputs, targets = next(data_iter)
print("Inputs:\n", inputs)
print("\nTargets:\n", targets)

the is no any error but when ongoing and running that code;
class SelfAttention_v1(nn.Module):

def __init__(self, d_in, d_out):
    super().__init__()
    self.d_out = d_out
    self.W_query = nn.Parameter(torch.rand(d_in, d_out))
    self.W_key   = nn.Parameter(torch.rand(d_in, d_out))
    self.W_value = nn.Parameter(torch.rand(d_in, d_out))
    
def forward(self, x):
    keys = x @ self.W_key
    queries = x @ self.W_query
    values = x @ self.W_value
    
    attn_scores = queries @ keys.T # omega
    attn_weights = torch.softmax(attn_scores / keys.shape[-1]**0.5, dim=-1)

    context_vec = attn_weights @ values
    return context_vec

torch.manual_seed(123)
sa_v1 = SelfAttention_v1(d_in, d_out)
sa_v1(inputs)

it gives me that error. And I convert inputs to float and then run the code above again then it did not give any error. ( I said before ı am very new at coding perhaps the problem might be due to me)

[rasbt]

Got it! Thanks for sharing!