Question Regarding LLMs Used in Training and Inference

[wz-wz111]

First, I would like to express my sincere appreciation for your excellent work. Your proposed method is highly innovative with a clear research approach, and it addresses important challenges. I found your paper very insightful and was deeply impressed by your contributions.

While studying your code implementation, I have a question regarding the model configuration that I hope you could help clarify:
Regarding the large language models used in training and inference:

• I noticed that the Gemini API is used during inference. However, if an API-based model is used during training, gradient backpropagation would not be possible.
• If different models are used for training and inference, does this discrepancy affect the performance of the knowledge adapter?

Understanding this aspect is crucial for me to fully grasp the working mechanism of your model. I would be extremely grateful if you could spare some time to address these questions.
Thank you once again for your outstanding work and for making your code publicly available!

[Mathews-Tom]

Thank you for your insightful question and kind words about the work. You've identified a key aspect of the architecture that deserves clarification.

How Training Works:

• The LLM is completely frozen during training (no gradient updates)
• Only the Knowledge Adapter parameters are trained
• Gradients flow through: Structural Embeddings → Fusion Layers → Projector
• The frozen LLM provides the loss signal (next-token prediction), but its weights never change

Paper's Intended Setup:

• Both training and inference use the same local open-source LLM (e.g., LLaMA-7B/8B)
• Soft prompts are embedding vectors injected into the LLM's input embedding space
• This allows the Knowledge Adapter to learn optimal representations for the target LLM

Current Implementation Note:
The repository's inference module currently uses Gemini API with text-based prompts, which differs from the paper's embedding-based approach. For full paper reproduction, inference should use the same local LLM with embedding injection as training.

Performance Impact of Model Mismatch:
Using different models would affect performance because the Projector is trained to map to a specific LLM's embedding space. The paper's plug-and-play claim (Table 5) requires retraining the adapter for each target LLM.

Update Coming Soon:
I'm actively working on updating the codebase to better align with the paper's methodology and provide clearer documentation on this architecture. Your feedback is valuable and will be addressed in an upcoming release.

[wz-wz111]

Thank you very much for your reply. Your explanation is very clear and has been very helpful in understanding the model's working mechanism. We look forward to your timely updates to the codebase to better align with the methodology of the paper, as well as to provide clearer architectural documentation. Your feedback is very valuable to us, and we will closely monitor the upcoming release. Thank you again for your excellent work!

[Mathews-Tom]

Update Released: LightPROF - Paper-Aligned Implementation

This issue has been addressed in the latest release. Key updates:

• LLaMA-3-8B-Instruct integration with soft prompt injection
• BERT-based Hop Predictor for 1-4 hop reasoning
• Knowledge Adapter with H+R-T structural encoding (per paper Eq. 4)
• Comprehensive test suite (113 tests passing)
• TOML configuration support

See PR #9 for details.

To update:
git pull origin main
uv sync

Documentation: https://github.com/Mathews-Tom/LightPROF#readme