Service fetch: variables are always string, allow access to data as list, dict etc.

[LaurensBurger]

Thema / Theme

Form designer

Omschrijving / Description

Currently all variables used in service fetch as query/header parameters are string, this does not allow a form builder to retrieve elements from list, dicts etc.

list = ["foo", "bar"]
When using {{ list.0 }} i expect "foo" in my query, but instead i get "["

Workaround currently is to first store list.0 in a separate string variable to be used as "foo" in my request.

[joeribekker]

Refinement:

This needs a stakeholder. We rely on feedback to make this feature better but are not going to do it on our own.

Small discussion about the use case for service fetch (SF):

Joeri: SF should be used a last resort; If a service/usage is crafted multiple times, this should evolve a separate plugin.
Sergei: Same feeling but thinks SF should be cleaned up and polished anyway.
Silvia: Same feeling, mostly worried about how it is used in practise (with great power comes great responsibility ... but not always taken).
Victorien and Vasileios are not familiar enough to share insights.

[sergei-maertens]

The fix to avoid localization while still allowing template expressions to post-process variables is too complicated to figure out in safe way. The inputs (template context) need to be escaped properly for the context where they will be used (request headers, query params or request URL), but that intrinsically causes str(...) to be called on a value and removes a lot of django template filter capacities (e.g. you can no longer take a datetime and apply a particular formatting through |date:"...".

However, only doing the escaping on the result of the template evaluation does also not work since unsafe characters can be injected and we can no longer see the individual bits that they're made up of.

This implementation is flawed because it was built with python string formatting in mind and not the django template engine, and this is amplified by some variables sometimes being a JSON type (e.g. a date in YYYY-MM-DD format, which is a string) or rich python types (like a date instance) that can be further processed in the template layer. The template syntax was retrofitted, but ultimately it's not a great fit and it creates localization problems too.

[sergei-maertens]

I had some time to let this sink in, and came to these conclusions:

• Service fetch has more issues than the templating aspect. Trying to solve this all at once isn't realistic and will require more funding & proper planning
• we must put the typing issues (str vs date) out of scope. We can now designate certain behaviours as unsupported
• the problem in this ticket is really blocking some workflows, so a bandaid fix for this is okay

To progress this, let's make the private _recursive_apply a public utility function. To address the str issues, add an extra argument: transform_leaf: boolean = False

You can then update service fetch to call recursive_apply(context, str, transform_leaf=True)

That allows users to access deep nested keys/lists, while making sure we can avoid data localization and can sanitize the data according to the usage context (query, header, path).

We then need to document that a limitation of service fetch is that its context variables are always strings.