Skip to content

b10242

Choose a tag to compare

View all tags
@github-actions github-actions released this 03 Aug 17:22
b10242
96278e3
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

CUDA: Add backend sampler for penalties sampler (#25262)

  • sampling: enhance penalty handling in common_sampler_init
  • Set default value for penalty_last_n based on model context if not specified.
  • Ensure penalty_last_n and n_prev are non-negative.
  • Update llama_sampler_penalties structure to inherit from llama_sampler_backend and add backend input handling for penalties.
  • Implement backend initialization and application logic for penalties, including frequency and presence adjustments.
  • tests: add backend penalties sampling tests and utility functions
  • Introduced accept_prompt and unique_prompt_tokens functions to handle prompt acceptance and token uniqueness.
  • Implemented compare_penalties_logits to compare logits from backend and CPU samplers with penalties.
  • Added test_backend_penalties_sampling to validate backend penalties with various configurations.
  • Enhanced the test suite for better coverage of penalty handling in sampling.

  • sampling: add support for top-k penalties in backend sampling

  • sampling: add fix to ensure stable numerical results. Preserve masked logits as -Inf and no longer generate NaN.

  • sampling: enhance penalty comparison tests with masking penalties logic

  • add comments on padding

  • sampling: add comments on modifications

  • add the unit test to cover masked-out token as -INF

  • validate repeat penalty to ensure it is finite and greater than 0; add tests for invalid values

  • refactor: test functions to share logic and be less verbose

  • add test to cover case where previously penalized token is not part of candidates

  • remove comments

  • remove redundant penalty_last_n initialization and validation in common_sampler_init

  • add support for penalties in sampler chain with configurable positions

  • add validation for penalty parameters and enhance tests for non-finite values

  • add context parameter to common_sampler_init and set default for penalty_last_n

  • add llama_n_ctx parameter to common_sampler_init for improved sampler initialization

  • replace penalty_last_n x n_candidates comparison matrix with a vocabulary-sized count tensor

  • add tests for backend penalties sampling without filler entries , token_count.size() == n_active == n_max == 64

  • add test for backend penalties sampling after top-p with large history window

  • remove as unused

  • add is_disabled method, tensor logits reshape, add rest review suggestions

  • clarify comment

Website:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI:

Assets 27
Loading