Exception when building HLG with large language model #132
Comments
drawfish
changed the title
|
Debugging this will require getting it in gdb and looking at the symbols on the stack. We need to trace back where the negative number came from. |
OK, I will try. |
Hi Dan，below is the stack info, it seem that the negative part came from the function: ElementSize(), but I don't know how to get the return value of ElementSize() from core dump file. |
|
ElementSize() should return 4, as a size_t. (However check that ElementSize() returns size_t in your code, your line numbers are different from mine). That should make a positive product as the 'int' should be promoted to size_t. |
In my code, the return type of ElementSize() is int32_t, just the same as the code in k2 master branch here. Either int32_t or size_t is OK. |
|
Ah yes, you are right. |
|
OK，I will continue to generate the decode graph after modifying my local code to ensure that no other erros occur. If everything goes well, I will make a pull request. |
|
After fixing the code above and then rerun the code, another exception occurs: |
|
|
Hm.
I'm sorry, that will be hard to fix.
The only real solution would be to have int64_t versions of all the
indexes, but we have chosen to keep all the indexes int32_t because the
size of everything doubles
if we make them all int64_t, and you can easily get OOM.
Your graph may simply be too large.
We support LM rescoring so you can always compile the HLG with a smaller LM
and then rescore with a larger one.
Can you make that PR though? That was a bug.
In the longer term I want to work with decoding algorithms that are more
memory-compact but can still use larger LMs, but that's in the future.
Dan
…On Mon, Nov 29, 2021 at 8:04 PM drawnfish ***@***.***> wrote:
Ah yes, you are right. It think this issue has been fixed in most places
already, but it seems to have somehow stayed in the codebase at this point.
Is there any chance you could make a pull request to fix this/
@danpovey <https://github.com/danpovey>
Hi, Dan. I think maybe the code in k2::IndexAxis0
<https://github.com/k2-fsa/k2/blob/8030001c9a002aa17e090a41de3f1146bdfe1e78/k2/csrc/ragged_ops.cu#L417>
needs to be totally modified. For example, I can be sure that 'Array2<int32_t>
old_offsets, new_offsets'
<https://github.com/k2-fsa/k2/blob/8030001c9a002aa17e090a41de3f1146bdfe1e78/k2/csrc/ragged_ops.cu#L431>
and 'Array1<int32_t> tot_sizes_out'
<https://github.com/k2-fsa/k2/blob/8030001c9a002aa17e090a41de3f1146bdfe1e78/k2/csrc/ragged_ops.cu#L437>
also have int32_t out-of-range problems.
But I can't fully understand the code involved in this function. I need
some help from your team.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#132 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAZFLOYB5BFLSDKHCLKPNZ3UONT3VANCNFSM5I2CMNAQ>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
|
Yh, at this stage, rescoring is the only solution. |
It seems that overflow occurs when building HLG on large language model.
The apra language model info:
After successfully detrminize(compose(L,G)) , an exception occurs when remove_epsilon from LG.
From the exception traceback we can see that something like overflow when cating impl_->byte_offset into int64_t ?
The code for compile HLG:
The text was updated successfully, but these errors were encountered: