Add a catflap to control HSH_Lookup from within the object iteration

[nigoroll]

If non-NULL, req->catflap_init is called before the cache lookup. It may set a catflap_f callback to be called for objcores and a catflap_fini_f to be called after the objcore iterator loop finishes.

A catflap_f may return any of enum catflap_e: FLP_CONTINUE to continue the loop (and get to look at the next object, if any), FLP_DEFAULT to fall through to the built-in default handling, FLP_MISS to force a miss or FLP_HIT to consider the objcore a hit.

A catflap_fini_f, if present, gets a final say on the object beauty contest. For example, a catflap_f may have saved the most beautiful object in the private pointer, but the decision is only final after all objects were considered. The return value of catflap_fini_f is either the object to be hit, or NULL to force a cache miss. The remaining decision on the details (e.g. if it was a HITMISS or HITPASS) happens as before.

An oc == NULL argument to catflap_fini_f signals that the catflap had been called on all objects, otherwise the iteration was terminated early.

Catflap chaining has deliberately been left out of the core implementation. vmods wishing to chain catflaps can still do so (by calling req->catflap_init and deploying a custom cat which knows how to
operate additional catflaps).

[nigoroll]

BTW, this is 50% of the open issues from #1799

[nigoroll]

updated, rebased and merged the aws alloc demo into a single commit

[hermunn]

I agree with the latest changes, and I think this is necessary to close #1799 in a satisfactory way.

[nigoroll]

bugwash:

• @nigoroll add VRT_Fail test
• @slimhazard suggested to move away from cat terminology
• @bsdphk to decide on this and the PR in general

[nigoroll]

FTR I am not entirely happy with 141ea56 for the following reasons:

• (commit message): The union was a suggestion by @hermunn which I added because he said it would be something @bsdphk would prefer. I am not happy about the fact that this is taken as an argument to reject the PR, in particular because my original commit had a void *
• Using multiple function pointers saves conditionals in the callbacks, which reduces the amount of work (and cpu pipeline stalls) under the oh mutex. Keeping the catflap efficient was a major concern from the start
• The init function can be replaced by additional work for the first call with state==0. But then even more conditionals are required under the oh mutex to avoid the init-work for additional ocs.
• The original proposal would have allowed for relatively simple catflap chaining: A catflap with chaining support could have saved the function and priv pointer to the next and called it if needed. With the reworked code I think chaining requires the last catflap in the chain to restore the first one's VCF

[bsdphk]

If @hermunn thinks I like unions, he should wonder why I put so few of them in Varnish :-)

Your concerns about efficiency and chaining are noted.

If real-world experience with this prototype-API indicates they are significant concerns, they will can be addressed, once we know what we need to fix.

But right now we are in "... no example at all." territory, so simplicity carries the day.