There is something with text and invariance.
The word 5 has a meaning in the sense that there are som invariants associated with it.
Let's say we have a vecotr space and an encoding mechanism. We know the encoding of 5. But the sentence 2 + 3 should be close to 5. And as humans, this fact is invariant though times. At least in our day to day life. Whatever exeprience I'm going to live, anyone sayong "2 + 3" should always map to 5. This means that each time I create a meaning-mapping, I add an invariant that my encoding mechanism must respect while evolving. Somehow, there is a dynamic loss for coherence though time.
Notice that the encoding of 5 can change as long as the encoding for 2+3 change in the same manner.
The same idea should come from every perception that can be mapped to 5 somehow. For example, if I focus on the number of things that I have in front of me, the resulting encoding should map to to the word 5 if there is 5 things in front of me.
We need dynamic embedding, we need to construct embedding sequentially. We can even create a curriculum for the embedding. Just as one can improve the precision of perception, you must be able to improve the precision of your embedding. You densify the embedding as you go while keeping invariants.
The following operations on embedding are needed:
- add a vector
- Consolidate duplicate vectors: it might happens that you believed 2 things were different but happens to be the same thing. (Notice that you actually remember the 2 things you took for the same thing even after recognising their sameness, so this reduce to add a vector and keeps 2 invariants)
the number of dimensions for words should probably be the same as the underlying "idea" space which is continuous. Words are just quantized idea, sentence are quantized idea, paragraph are quantized idea etc.
Association are just the capacity to recognise k-nearest neighbours from a resulting vector.
One might also need an embedding of functions.