Clarification : Fixed feature vectors

[astariul]

Please correct me if I'm wrong :

• Feature vectors are word embeddings, for each token of the input file.
• These vectors can be used as ELMo / GloVe : as a base for a bigger neural network.

If these assumptions are right, here is my question :

From the use example :

python extract_features.py
...
--layers=-1,-2,-3,-4
...

Why would anyone be interested in features vectors from others layers than the last one ?

From my understanding, feature vectors from the last layer are complete. Feature vectors from other layers are not complete.

'Complete' is obviously the wrong word here, due to my lack of vocabulary / knowledge.

By the way, BERT is really amazing, congratulations and thank you for sharing it.

[cbockman]

See section 5.4 in the paper: https://arxiv.org/pdf/1810.04805.pdf

tldr; better results from grabbing last 4 than just last 1.