How to control feature input strategies in the released code?

[tianzhangwu]

In section 3.4 of the paper, it states:

“We validate multiple feature input strategies: (i) Feature from all scales (ii) Final-scale feature only (iii) Residual features from all scales. We discover that features from the final scale achieves best overall performance, as detailed in Appendix B.1.”

It there a config option for controlling the feature input strategies? It seems that the current setup always uses features from all scales.

Thanks you!

[QuLiao1117]

@tianzhangwu Thanks for your attention on TokenFlow.

For Table 8, take 256x256 resolution as example, "full scale" means feeding all tokens to the LLM, including tokens from all scales (1x1, 2x2, ..., 16x16). While "last scale" only uses the last 16*16 as input.

The code currently defaults to using "last scale" setting. If you want to experiment with other configurations, you can modify the encode function to extract different feature combinations as needed.

[tianzhangwu]

@tianzhangwu Thanks for your attention on TokenFlow.

For Table 8, take 256x256 resolution as example, "full scale" means feeding all tokens to the LLM, including tokens from all scales (1x1, 2x2, ..., 16x16). While "last scale" only uses the last 16*16 as input.

The code currently defaults to using "last scale" setting. If you want to experiment with other configurations, you can modify the encode function to extract different feature combinations as needed.

but it seems that the returned final_quantize actually adds results from all layers，with the following code
final_quantize += quantize

[tianzhangwu]

https://github.com/ByteFlow-AI/TokenFlow/blob/d50b14dc0e873623a0ee263a4bca8397d2d52411/tokenflow/tokenizer/vq_model.py#L330

[QuLiao1117]

@tianzhangwu Thanks for your attention on TokenFlow.
For Table 8, take 256x256 resolution as example, "full scale" means feeding all tokens to the LLM, including tokens from all scales (1x1, 2x2, ..., 16x16). While "last scale" only uses the last 16*16 as input.
The code currently defaults to using "last scale" setting. If you want to experiment with other configurations, you can modify the encode function to extract different feature combinations as needed.

but it seems that the returned final_quantize actually adds results from all layers，with the following code final_quantize += quantize

Yes, that means the "last scale" in this table. Using only the "last residual feature" would result in worse performance.

[tianzhangwu]

@tianzhangwu Thanks for your attention on TokenFlow.
For Table 8, take 256x256 resolution as example, "full scale" means feeding all tokens to the LLM, including tokens from all scales (1x1, 2x2, ..., 16x16). While "last scale" only uses the last 16*16 as input.
The code currently defaults to using "last scale" setting. If you want to experiment with other configurations, you can modify the encode function to extract different feature combinations as needed.

but it seems that the returned final_quantize actually adds results from all layers，with the following code final_quantize += quantize

Yes, that means the "last scale" in this table. Using only the "last residual feature" would result in worse performance.

Thank you for the clarification. However, I am still a bit confused about the terminology. What exactly does "Full scale" mean if "last scale" and "last residual feature" are not the same?

Doesn't the table from VAR suggest that "last scale" and "last residual feature" are the same?

[tianzhangwu]

Oh, I see, by "Full scale" you might mean the concatenation of all quantized features.

[QuLiao1117]

"Full scale" refers to utilizing tokens from all available scale levels. Using TokenFlow-B as an example, the scale settings are [1,2,4,6,8,10,12,14]. In full scale mode, it inputs 561 (1 + 4 + 16 + 36 + 64 + 100 + 144 + 196 = 561) tokens to LLaVA.
Each scale's tokens represent accumulated features. This approach was tested to verify whether lower-level coarse features could contribute additional high-level semantic meaning, potentially improving overall multimodal understanding performance. However, the results demonstrated that no improvement was achieved, indicating that the last accumulated feature (196 tokens) is sufficient for understanding.

Regarding the VAR pseudocode, it describes the process of obtaining indices for all scales. For next-scale prediction, they use idxBl_to_var_input function to get the accumulated features. In our t2i implementation, we use similar code to achieve the same functionality. In i2t scenario, using only the last scale is sufficient to achieve optimal results.

[tianzhangwu]

"Full scale" refers to utilizing tokens from all available scale levels. Using TokenFlow-B as an example, the scale settings are [1,2,4,6,8,10,12,14]. In full scale mode, it inputs 561 (1 + 4 + 16 + 36 + 64 + 100 + 144 + 196 = 561) tokens to LLaVA. Each scale's tokens represent accumulated features. This approach was tested to verify whether lower-level coarse features could contribute additional high-level semantic meaning, potentially improving overall multimodal understanding performance. However, the results demonstrated that no improvement was achieved, indicating that the last accumulated feature (196 tokens) is sufficient for understanding.

Regarding the VAR pseudocode, it describes the process of obtaining indices for all scales. For next-scale prediction, they use idxBl_to_var_input function to get the accumulated features. In our t2i implementation, we use similar code to achieve the same functionality. In i2t scenario, using only the last scale is sufficient to achieve optimal results.

Thanks ！