Substitution in <dyn_variable> for JSONPath

[tisba]

Proposal for #316

This PR propagates the substitution flag from <request subst="…"> into the dynvar specification used in ts_search:parse_dynvar. NOTE Only jsonpath has been adjusted currently as they don't need to compile. The performance impact should be minimal.

It was necessary to pass the current DynVars into ts_search:parse_dynvar so the arity was changed from 2 to 3. For compatibility reasons (tests mainly) ts_search:parse_dynvar/2 is now also present with the same behaviour as before. ts_search:parse_dynvar/3 is now used by ts_client:update_stats and it is the main point of entry.

This PR basically allows us to have this working, which works a lot more as expected to the user:

<request subst="true">
  <dyn_variable name="title1" jsonpath="$.slideshow.slides[?title==%%_title%%].type"/>
  <http url="https://httpbin.org/json" method="GET" version="1.1">
  </http>
</request>

Note, that the substitution is only performed, when the subst attribute is set to true and we detect a variable inside the expression (%%*%%).

Example

<setdynvars sourcetype="value" value="Overview">
  <var name="title"/>
</setdynvars>

<!-- substitutions are enabled and actually activated, because %%*%% was detected -->
<request subst="true">
  <dyn_variable name="title1" jsonpath="$.slideshow.slides[?title==%%_title%%].type"/>
  <http url="https://httpbin.org/json" method="GET" version="1.1">
  </http>
</request>

<!-- nothing will be substitute, the jsonpath is taken literally -->
<request>
  <dyn_variable name="title2" jsonpath="$.slideshow.slides[?title==%%_title%%].type"/>
  <http url="https://httpbin.org/json" method="GET" version="1.1">
  </http>
</request>

<!-- substitutions are enabled but there isn't anything to substitute (internally no sub ist attempted) -->
<request subst="true">
  <dyn_variable name="title3" jsonpath="$.slideshow.title"/>
  <http url="https://httpbin.org/json" method="GET" version="1.1">
  </http>
</request>

JSON document returned by https://httpbin.org/json:

{
    "slideshow": {
        "author": "Yours Truly",
        "date": "date of publication",
        "slides": [{
                "title": "Wake up to WonderWidgets!",
                "type": "all"
            },
            {
                "items": [
                    "Why <em>WonderWidgets</em> are great",
                    "Who <em>buys</em> WonderWidgets"
                ],
                "title": "Overview",
                "type": "all"
            }
        ],
        "title": "Sample Slide Show"
    }
}

Will produce:

ts_search:(6:<0.158.0>) Dyn Var: Match (title1=<<"all">>),
…
ts_search:(5:<0.158.0>) Dyn Var: no Match (varname=title2),
…
ts_search:(6:<0.158.0>) Dyn Var: Match (title3=<<"Sample Slide Show">>),
…

How to proceed?

1. I'd like to see if this approach is desirable or not
2. I'd like to propose to start first with only one type of substitution. Mainly because other types require pre-compilation of the used expression where we probably want to be a bit more careful.

[tisba]

I'd really like to hear your feedback, @nniclausse. Mainly if this approach is at all desirable or not. And if not, if you have another idea what approach you'd see to realise such a enhancement.

[tisba]

@nniclausse ping. I'd really like your feedback. We're using the current patch for some time now and so far it did work just fine.

My idea would be to merge this now, and create follow-up issues that we also apply substitutions to other dyn var types. They might have a perf impact, as regular expressions and xpaths will require recompilation. But IMO this is okay, if we document this properly so that the users are aware of this.

[nniclausse]

I'm OK with the general idea. Appart from the small typo, the current PR lacks some documentation and maybe some unit tests ?

[nniclausse]

small typo to fix here: should be EnableSubstitution

[tisba]

fixed

[tisba]

okay great. I'll take a look at how to test this and will add some remarks to the documentation.

[tisba]

Do you mind to take another look, @nniclausse? I added some tests and a paragraph to the JSONpath documentation. I'm not sure how to go about mentioning unreleased versions. Is it okay to just assume that this will be in 1.8.0?

[nniclausse]

Yes, that's ok, thanks !