More details about the "gather embeddings"?

[asheroin]

It would be nice if could give more details about the gather embeddings ? From the image from README.md, it seems that just simple concat the clip embeddings but still got an shape of 768*77, just same as the output from hide all tokens's.

[hnmr293]

For example, use the following prompt and target tokens. For clarity, I've also added the token number to the prompt.

prompt:    a cute girl, blue eyes, red shirt
token idx: 1 2~~~ 3~~~4 5~~~ 6~~~7 8~~ 9~~~~
targets:   blue, red

NB. The comma is an independent token.

For this prompt and target tokens, Cutoff (this extension) generates the three prompts shown below.

token idx $t$	token	applied prompt	prompt id $p$
0	<|startoftext|>	a cute girl, blue eyes, red shirt	0
1	a	a cute girl, blue eyes, red shirt	0
2	cute	a cute girl, blue eyes, red shirt	0
3	girl	a cute girl, blue eyes, red shirt	0
4	,	a cute girl, blue eyes, red shirt	0
5	blue	a cute girl, _ eyes, red shirt	1
6	eyes	a cute girl, _ eyes, red shirt	1
7	,	a cute girl, blue eyes, red shirt	0
8	red	a cute girl, blue eyes, _ shirt	2
9	shirt	a cute girl, blue eyes, _ shirt	2
10	<|endoftext|>	a cute girl, blue eyes, red shirt	0

CLIP converts these three prompts into three corresponding vectors (each a 77x768 matrix):

prompt id $p$	77 vectors of 768 dimensions
0	$( \quad v^0_0 \quad v^0_1 \quad v^0_2 \quad ... \quad v^0_{76} \quad )$
1	$( \quad v^1_0 \quad v^1_1 \quad v^1_2 \quad ... \quad v^1_{76} \quad )$
2	$( \quad v^2_0 \quad v^2_1 \quad v^2_2 \quad ... \quad v^2_{76} \quad )$

where $v^p_t$ is a 768-dimensional vector which represents the $t$-th token of the prompt $p$.

For instance, $v^0_1, v^1_1, v^2_1$ are the vectors corresponding to the token a in prompts $p=0,1,2$.
Likewise, $v^0_2, v^1_2, v^2_2$ are the vectors corresponding to the token cute in prompts $p=0,1,2$.
The same applies to other $v^p_t$.

From the above, we have a vector to be applied to each token.

token idx $t$	token	applied prompt	prompt id $p$	vector to be applied
0	<|startoftext|>	a cute girl, blue eyes, red shirt	0	$v^0_0$
1	a	a cute girl, blue eyes, red shirt	0	$v^0_1$
2	cute	a cute girl, blue eyes, red shirt	0	$v^0_2$
3	girl	a cute girl, blue eyes, red shirt	0	$v^0_3$
4	,	a cute girl, blue eyes, red shirt	0	$v^0_4$
5	blue	a cute girl, _ eyes, red shirt	1	$v^1_5$
6	eyes	a cute girl, _ eyes, red shirt	1	$v^1_6$
7	,	a cute girl, blue eyes, red shirt	0	$v^0_7$
8	red	a cute girl, blue eyes, _ shirt	2	$v^2_8$
9	shirt	a cute girl, blue eyes, _ shirt	2	$v^2_9$
10	<|endoftext|>	a cute girl, blue eyes, red shirt	0	$v^0_{10}$

The embedding to be applied is:
$$( \quad v^0_0 \quad v^0_1 \quad v^0_2 \quad v^0_3 \quad v^0_4 \quad v^1_5 \quad v^1_6 \quad v^0_7 \quad v^2_8 \quad v^2_9 \quad v^0_{10} \quad v^0_{11} \quad ... \quad v^0_{76} \quad )$$

Sorry for my long text. I used the word "gather" with the intention of the action of picking one specific vector for each $t$ from the $p$ prompts in this way.

[asheroin]

Just still a little confused about the term Cutoff. In this case, you actually cut off the current token. However, from the overall discription, it seem to cut off the other words that may affect the current one.

[marson666]

i print the log for Cutoff, the "gather embeddings" method is clear explained by the @hnmr293 , i think the cut off words should be the way in the log, as @asheroin said

below is the log info

a cute girl, white hair, _ shirt, _ tie, _ shoes, _ short pants, _ eyes
005 white a cute girl, white hair, _ shirt, _ tie, _ shoes, _ short pants, _ eyes
006 hair a cute girl, white hair, _ shirt, _ tie, _ shoes, _ short pants, _ eyes

a cute girl, _ hair, black shirt, _ tie, _ shoes, _ short pants, _ eyes
008 black a cute girl, _ hair, black shirt, _ tie, _ shoes, _ short pants, _ eyes
009 shirt a cute girl, _ hair, black shirt, _ tie, _ shoes, _ short pants, _ eyes

a cute girl, _ hair, _ shirt, green tie, _ shoes, _ short pants, _ eyes
011 green a cute girl, _ hair, _ shirt, green tie, _ shoes, _ short pants, _ eyes
012 tie a cute girl, _ hair, _ shirt, green tie, _ shoes, _ short pants, _ eyes

a cute girl, _ hair, _ shirt, _ tie, red shoes, _ short pants, _ eyes
014 red a cute girl, _ hair, _ shirt, _ tie, red shoes, _ short pants, _ eyes
015 shoes a cute girl, _ hair, _ shirt, _ tie, red shoes, _ short pants, _ eyes

a cute girl, _ hair, _ shirt, _ tie, _ shoes, blue short pants, _ eyes
017 blue a cute girl, _ hair, _ shirt, _ tie, _ shoes, blue short pants, _ eyes
018 short a cute girl, _ hair, _ shirt, _ tie, _ shoes, blue short pants, _ eyes
019 pants a cute girl, _ hair, _ shirt, _ tie, _ shoes, blue short pants, _ eyes

a cute girl, _ hair, _ shirt, _ tie, _ shoes, _ short pants, yellow eyes
021 yellow a cute girl, _ hair, _ shirt, _ tie, _ shoes, _ short pants, yellow eyes
022 eyes a cute girl, _ hair, _ shirt, _ tie, _ shoes, _ short pants, yellow eyes

[hnmr293]

Cutoff is the name of the concept. The implementation details described above are not directly related to the extension name. Sorry for the confusion caused by my poor explanation.

Note.
This is a picture of what I initially had in mind, and I named this extension Cutoff from this image.

And this is the result when I confirmed the concept. I've developed EvViz2 for this purpose.

[hnmr293]

@marson666 that's right. Enabling Debug log will help you to understand implementation, @asheroin.

[Cutoff] replace: white, green, red, blue, yellow, pink
[Cutoff] to: _</w> (318)
[Cutoff] original: a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
[Cutoff]       #0: a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
[Cutoff]       #1: a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes,  _  skirt
[Cutoff]       #2: a cute girl, white shirt with green tie, red shoes, blue hair,  _  eyes, pink skirt
[Cutoff]       #3: a cute girl, white shirt with green tie, red shoes, blue hair,  _  eyes,  _  skirt
[Cutoff]       #4: a cute girl, white shirt with green tie, red shoes,  _  hair, yellow eyes, pink skirt
[Cutoff]       #5: a cute girl, white shirt with green tie, red shoes,  _  hair, yellow eyes,  _  skirt
[Cutoff]       #6: a cute girl, white shirt with green tie, red shoes,  _  hair,  _  eyes, pink skirt
[Cutoff]       #7: a cute girl, white shirt with green tie, red shoes,  _  hair,  _  eyes,  _  skirt
[Cutoff]       #8: a cute girl, white shirt with green tie,  _  shoes, blue hair, yellow eyes, pink skirt
[Cutoff]       #9: a cute girl, white shirt with green tie,  _  shoes, blue hair, yellow eyes,  _  skirt
[Cutoff]       #10: a cute girl, white shirt with green tie,  _  shoes, blue hair,  _  eyes, pink skirt
[Cutoff]       #11: a cute girl, white shirt with green tie,  _  shoes, blue hair,  _  eyes,  _  skirt
[Cutoff]       #12: a cute girl, white shirt with green tie,  _  shoes,  _  hair, yellow eyes, pink skirt
[Cutoff]       #13: a cute girl, white shirt with green tie,  _  shoes,  _  hair, yellow eyes,  _  skirt
[Cutoff]       #14: a cute girl, white shirt with green tie,  _  shoes,  _  hair,  _  eyes, pink skirt
[Cutoff]       #15: a cute girl, white shirt with green tie,  _  shoes,  _  hair,  _  eyes,  _  skirt
[Cutoff]       #16: a cute girl,  _  shirt with  _  tie, red shoes, blue hair, yellow eyes, pink skirt
[Cutoff]       #17: a cute girl,  _  shirt with  _  tie, red shoes, blue hair, yellow eyes,  _  skirt
[Cutoff]       #18: a cute girl,  _  shirt with  _  tie, red shoes, blue hair,  _  eyes, pink skirt
[Cutoff]       #19: a cute girl,  _  shirt with  _  tie, red shoes, blue hair,  _  eyes,  _  skirt
[Cutoff]       #20: a cute girl,  _  shirt with  _  tie, red shoes,  _  hair, yellow eyes, pink skirt
[Cutoff]       #21: a cute girl,  _  shirt with  _  tie, red shoes,  _  hair, yellow eyes,  _  skirt
[Cutoff]       #22: a cute girl,  _  shirt with  _  tie, red shoes,  _  hair,  _  eyes, pink skirt
[Cutoff]       #23: a cute girl,  _  shirt with  _  tie, red shoes,  _  hair,  _  eyes,  _  skirt
[Cutoff]       #24: a cute girl,  _  shirt with  _  tie,  _  shoes, blue hair, yellow eyes, pink skirt
[Cutoff]       #25: a cute girl,  _  shirt with  _  tie,  _  shoes, blue hair, yellow eyes,  _  skirt
[Cutoff]       #26: a cute girl,  _  shirt with  _  tie,  _  shoes, blue hair,  _  eyes, pink skirt
[Cutoff]       #27: a cute girl,  _  shirt with  _  tie,  _  shoes, blue hair,  _  eyes,  _  skirt
[Cutoff]       #28: a cute girl,  _  shirt with  _  tie,  _  shoes,  _  hair, yellow eyes, pink skirt
[Cutoff]       #29: a cute girl,  _  shirt with  _  tie,  _  shoes,  _  hair, yellow eyes,  _  skirt
[Cutoff]       #30: a cute girl,  _  shirt with  _  tie,  _  shoes,  _  hair,  _  eyes, pink skirt
[Cutoff]       #31: a cute girl,  _  shirt with  _  tie,  _  shoes,  _  hair,  _  eyes,  _  skirt
000 <|startoftext|>  a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
001 a</w>            a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
002 cute</w>         a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
003 girl</w>         a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
004 ,</w>            a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
010 ,</w>            a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
013 ,</w>            a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
016 ,</w>            a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
019 ,</w>            a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
022 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
023 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
024 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
025 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
026 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
027 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
028 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
029 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
030 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
031 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
032 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
033 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
034 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
035 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
036 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
037 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
038 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
039 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
040 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
041 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
042 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
043 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
044 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
045 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
046 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
047 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
048 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
049 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
050 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
051 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
052 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
053 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
054 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
055 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
056 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
057 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
058 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
059 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
060 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
061 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
062 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
063 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
064 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
065 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
066 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
067 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
068 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
069 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
070 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
071 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
072 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
073 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
074 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
075 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
076 <|endoftext|>    a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
005 white</w>        a cute girl, white shirt with green tie,  _  shoes,  _  hair,  _  eyes,  _  skirt
006 shirt</w>        a cute girl, white shirt with green tie,  _  shoes,  _  hair,  _  eyes,  _  skirt
007 with</w>         a cute girl, white shirt with green tie,  _  shoes,  _  hair,  _  eyes,  _  skirt
008 green</w>        a cute girl, white shirt with green tie,  _  shoes,  _  hair,  _  eyes,  _  skirt
009 tie</w>          a cute girl, white shirt with green tie,  _  shoes,  _  hair,  _  eyes,  _  skirt
011 red</w>          a cute girl,  _  shirt with  _  tie, red shoes,  _  hair,  _  eyes,  _  skirt
012 shoes</w>        a cute girl,  _  shirt with  _  tie, red shoes,  _  hair,  _  eyes,  _  skirt
014 blue</w>         a cute girl,  _  shirt with  _  tie,  _  shoes, blue hair,  _  eyes,  _  skirt
015 hair</w>         a cute girl,  _  shirt with  _  tie,  _  shoes, blue hair,  _  eyes,  _  skirt
017 yellow</w>       a cute girl,  _  shirt with  _  tie,  _  shoes,  _  hair, yellow eyes,  _  skirt
018 eyes</w>         a cute girl,  _  shirt with  _  tie,  _  shoes,  _  hair, yellow eyes,  _  skirt
020 pink</w>         a cute girl,  _  shirt with  _  tie,  _  shoes,  _  hair,  _  eyes, pink skirt
021 skirt</w>        a cute girl,  _  shirt with  _  tie,  _  shoes,  _  hair,  _  eyes, pink skirt

[NormXU]

Correct me if I am wrong, but according to the debug log shown by @marson666, the vector tables should be like

token idx  t	token	applied prompt	prompt id p	vector to be applied
0	<|startoftext|>	a cute girl, blue eyes, red shirt	0	$v_0^0$
1	a	a cute girl, blue eyes, red shirt	0	$v_1^0$
2	cute	a cute girl, blue eyes, red shirt	0	$v_2^0$
3	girl	a cute girl, blue eyes, red shirt	0	$v_3^0$
4	,	a cute girl, blue eyes, red shirt	0	$v_4^0$
5	blue	a cute girl, blue eyes, _ shirt	2	$v_5^2$
6	eyes	a cute girl, blue eyes, _ shirt	2	$v_6^2$
7	,	a cute girl, blue eyes, red shirt	0	$v_7^0$
8	red	a cute girl, _ eyes, red shirt	1	$v_8^1$
9	shirt	a cute girl, _ eyes, red shirt	1	$v_9^1$
10	<|endoftext|>	a cute girl, blue eyes, red shirt	0	$v_{10}^0$

[juxuan27]

I agree with @NormXU . However, do that mean the explaination from the author @hnmr293 of this repo is incorrecnt? A little bit cofused hhh.