fix embedding encode

[tastelikefeet]

PR type

• Bug Fix
• New Feature
• Document Updates
• More Models or Datasets Support

PR information

Write the detail information belongs to this PR.

Experiment results

Paste your experiment result here(if needed).

[gemini-code-assist]

Summary of Changes

Hello @tastelikefeet, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a bug in the _embedding_encode function within swift/llm/template/base.py. The core issue was an inconsistency in how negative examples were encoded, specifically when different negative inputs contained varying sets of keys. The fix refactors the encoding process to ensure that all generated embedding lists for negative examples maintain uniform lengths, padding with None where necessary, thereby preventing potential data misalignment or errors in downstream processing.

Highlights

• Refactored Negative Embedding Encoding: The logic for encoding negative examples within the _embedding_encode function has been significantly refactored to ensure data consistency.
• Consistent List Lengths: The updated implementation now guarantees that all generated lists for negative_{key} embeddings maintain uniform lengths, crucial for downstream processing.
• Handling Missing Keys: New logic has been introduced to correctly handle scenarios where different negative examples might have varying sets of keys, padding with None where a key is absent for a particular example.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request fixes a bug in the _embedding_encode method where encoded features from negative samples with varying keys could become misaligned. The new logic correctly pads missing features with None, ensuring all feature lists for negative samples maintain consistent lengths. This is a solid fix. I've provided one suggestion to refactor the implementation for improved readability and maintainability.

[gemini-code-assist]

The logic to handle potentially different sets of keys from negative samples is correct. However, it can be refactored for better readability and maintainability. The current implementation is a bit dense. A clearer approach would be to first identify the keys for the current negative sample, then handle padding for both existing and new keys in separate, more explicit steps.

Suggested change

	for idx, negative in enumerate(inputs.negative):
	_tmp_negative_keys = set()
	negative_encoded = self._encode_truncated(negative)
	for key in negative_encoded:
	if f'negative_{key}' not in _encoded:
	_encoded[f'negative_{key}'] = []
	_encoded[f'negative_{key}'].append(negative_encoded[key])
	negative_key = f'negative_{key}'
	_all_negative_keys.add(negative_key)
	_tmp_negative_keys.add(negative_key)
	if negative_key not in _encoded:
	_encoded[negative_key] = [None] * idx
	_encoded[negative_key].append(negative_encoded[key])
	for miss_key in (_all_negative_keys - _tmp_negative_keys):
	_encoded[miss_key].append(None)
	labels.append(0.0)
	for idx, negative in enumerate(inputs.negative):
	negative_encoded = self._encode_truncated(negative)
	current_negative_keys = {f'negative_{k}' for k in negative_encoded}
	# Pad for keys seen in other samples but not in the current one
	missing_keys = _all_negative_keys - current_negative_keys
	for key in missing_keys:
	_encoded[key].append(None)
	# Add keys from the current sample
	for key, value in negative_encoded.items():
	negative_key = f'negative_{key}'
	if negative_key not in _encoded:
	# New key, pad for previous samples
	_encoded[negative_key] = [None] * idx
	_encoded[negative_key].append(value)
	_all_negative_keys.update(current_negative_keys)
	labels.append(0.0)