Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix embedding copy size #1036

Merged
merged 3 commits into from
Feb 18, 2024
Merged

fix embedding copy size #1036

merged 3 commits into from
Feb 18, 2024

Conversation

irexyc
Copy link
Collaborator

@irexyc irexyc commented Jan 24, 2024

Motivation

embedding 的拷贝逻辑有两个问题:

  1. 靠后的embedding不需要拷贝
  2. byte_size 的计算有问题,因为不一定拷贝完整的embedding

考虑下面的情况

cacheed: C, length=7
h_input: X, length=5
normal token: A
embedding token: B
embedding offset: [4, 9), [12, 16)

CCCCCXXXXXXX        // cached + h_input
AAAABBBBBAAABBBBAAA // context

针对1), 第二个embedding, 7 + 5 - 1 < 12,所以不需要拷贝
针对2), 要更新 begin, end. begin = max(4, 5) = 5; (end - begin) = 4, 拷贝4个元素

@irexyc irexyc added the WIP label Jan 24, 2024
@irexyc irexyc removed the WIP label Jan 24, 2024
@lvhan028 lvhan028 self-requested a review January 30, 2024 13:31
@lvhan028 lvhan028 merged commit 1546b0b into InternLM:main Feb 18, 2024
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants