Delayed expression lookup during generation

[linas]

@ampli During generation, I would like to delay expression lookup for wild-cards, until after pruning. For example, this is \* results in

   +-Ss--+---(A+ or B+ or C+)
   |     |
this.p is.v  

and then I look in the dict only for those words that have (A+ or B+ or C+) on them.

The simplest way I can think of hacking this is to create a category with every possible link in it, and then, outside of LG, select only those words that have the right connector.

For more complicated things, I would want

    +-----------------------------(A+ or B+ or C+)
    +----------->Wd------------+
    |      +-------Ds**c-------+
    |      |        +-----A----+--(A+ or B+ or C+)
    |      |        |          |
LEFT-WALL the complicated.a thing.n

which could be obtained with the complicated thing \* \* which would allow a wild-card category where each disjunct is just a single connector ... this is a \* \*

    +----->WV----->+-------(A+ or B+ or C+)
    +-->Wd---+-Ss*b+     +---(A+ or B+ or C+)
    |        |     |     |
LEFT-WALL this.p is.v    a

Is there some better way to do this? I fear the case where there might be hundreds of thousands of link types.

Right now, I'm only thinking about tacking these on the the end of a phrase that has already started, and not in the middle

[ampli]

and then I look in the dict only for those words that have (A+ or B+ or C+) on them.

Is this for speedup or another purpose?

In case it is for speedup:
There are several ways to speed things up; one is to have a "frozen file" containing a binary representation of the expression of all the categories. If you load it at the place of * and then expression_prune() (which is fast), only the expressions of the words that may connect would remain.

to create a category with every possible link in it

This is currently done when creating the expression for all the categories.
For sentences that contain mainly fixed strings, expression_prune() promptly removes most of it. It may be beneficial to include power pruning in expression_prune() to make it even more effective (this will save time in build_disjuncts() and power_prune()).

[linas]

My code crashed, because I had zero categories in the dict.

As I write this reply, I now begin to see the solution, and so perhaps the above sketch is not quite correct.

This is the word-pairs-only dict; let me explain how it works. The underlying dataset contains just word-pairs, with costs. Of course, LG, needs disjuncts, not pairs, and so during lookup_dict() the disjuncts are created on the fly, out of 1,2,3 or 4 pairs (configurable) Think of each word-pair as a distinct connector type. Such auto-creation has a combinatorial explostion, because if a word is in $N$ pairs, then there are $N$ single-connector disjuncts ... and $N^2$ double-conector disjuncts, etc. So this gets out of control, quick.

To avoid this, I pre-tokenize the sentence, and only consider word-pairs between words that are actually in the sentence. This keeps $N$ small: it can never be larger than the sentence length. This works great for ordinary parsing.

For generation, I would have to ... ??? ... One possibility is to create one category for each possible word-pair that might link to some word in the sentence. The number $N$ of such single-connector categories could be large, and so the number $N^2$ double-connector categories is worse.

The other possibility is to create a dummy link type: ACY that can connect to anything, and have a category with just ACY- or (ACY- & ACY-) or (ACY- & ACY- & ACY-) in it. Run the parser, and then select an appropriate word later. This works... although it has an issue

Different word-pairs have different costs, and using ACY as a dummy connector hides this difference. So after selecting a word, the total cost has to be adjusted. Well, I guess that's doable, that is indeed how its supposed to work ... Hmm.

So I see how to solve this problem, now. Onwards and upwards. Thank you for your time!

(Explaining the problem to someone else often helps find a new route to a solution. This one is interesting, because ... well, I'll save that for some other day.)

[whitten]

Linas,
this is your bailiwick, but reading this I wonder if creating a series of dummy link type like ACY but with associated costs would work such as ACY-COST-0, ACY-COST-1, ACY-COST-2, ACY-COST-3 etc. would make the cost of the substitutable word-pair costs be transparent at the time of the selection. This would still provide a link that could be replaced, but would do it at the cost of a messy set of categories (exponential ?)

Dave

[linas]

Wow! That's a great idea! My costs are floating-point numbers, but putting them into buckets, say 100 buckets, is manageable. So for $N=100$ then $N^3=1M$ which is near the boundary of what LG can do in a reasonable amount of time (it blows up once more than millions of things get involved). Even 10 or 20 buckets would eliminate some of the ugliest stuff. I could even have fine-grained costs if there is only one link, and course-grained when there are many. Cool! I like the idea! Now, to code it up...

I'm suppressing the urge to blabber about the down-stream stuff. I'm thrilled to have you reading through this, @whitten