PCC: Do you add end of sentence tokens to the target ids?

[Darkbblue]

Good work there! I've read the Decoder code and the pre-training data preprocessing code, and am a bit confused right now. It seems that neither codes add an token to the target id list. Is this intended? I thought this should be necessary for correct training.

[broalantaps]

@Darkbblue Thank you for your attention!
You may refer to utils.py

if self.stage == 1:
      llm_ids = [text["llm_ids"] + [self.llm_eos_token_id] for text in examples]
      next_ids = [text["next_ids"] + [self.llm_eos_token_id] for text in examples]

Please feel free to reach out if you have any further questions!

[Darkbblue]

@Darkbblue Thank you for your attention! You may refer to utils.py

if self.stage == 1:
      llm_ids = [text["llm_ids"] + [self.llm_eos_token_id] for text in examples]
      next_ids = [text["next_ids"] + [self.llm_eos_token_id] for text in examples]

Please feel free to reach out if you have any further questions!

Thanks! Now I know where to find the eos tokens. BTW, I have another question: In the paper, you have experimented with different compression rates. Do you train a model separately for each different rate, so the model can only work under this specific rate? Or can a single model be trained to work with different rates by, like, dynamically sampling a rate from several rates for each step?

[broalantaps]

@Darkbblue This is a very good question. I trained a separate model for each different compression rate. I think this will yield better results, and I believe a general-purpose model should indeed be able to handle different compression rates, so adopting adaptive compression rates based on the input text is an import direction for the future.

[Darkbblue]

@Darkbblue This is a very good question. I trained a separate model for each different compression rate. I think this will yield better results, and I believe a general-purpose model should indeed be able to handle different compression rates, so adopting adaptive compression rates based on the input text is an import direction for the future.

Thanks for the reply. Then, I suppose dynamic compression rate is not a thing that can be reliably implemented currently. There might be some difficulties in it to be uncovered. I will take this into consideration in my own study.

[broalantaps]

@Darkbblue I suppose only RL can achieve it. If you have any other questions, feel free to reopen this issue.