Feature Request: Allow datasources to be defined dynamically

[osterman]

what

• Provide a getJSON and getYAML functions

why

• Very common format for APIs and source files
• Inline interpolation
• The current --data parameter is insufficient for dynamic use cases (e.g. range over a list); it requires the caller to know in advance all the data endpoints

use-cases

• include data by calling github APIs.
• Include data from remote data files like this: https://github.com/cloudposse/geodesic/blob/master/README.yaml

Related

The immensely popular hugo CMS framework provides this method.

• https://gohugo.io/templates/data-templates/#call-the-functions-with-a-url

[hairyhenderson]

@osterman I've had thoughts in the past about being able to define datasources inline, which may be a good general-case way to achieve what you're looking for.

Something like:

{{ $d := datasource "https://github.com/cloudposse/geodesic/blob/master/README.yaml" }}
...

The idea is that the datasource function would recognize the first argument (which is supposed to be the datasource alias) as non-existant, but also as a URL, so it would allow it and dynamically add it to the list of datasources.

Another approach, which may help with re-use:

{{ $d := datasource "somealias=https://github.com/cloudposse/geodesic/blob/master/README.yaml" }}
...
{{ datasource "somealias" }} <- this now refers to the same datasource

Or it may even make more sense to not overload the datasource function and introduce a new function:

{{ defineDatasource "somealias" "https://github.com/cloudposse/geodesic/blob/master/README.yaml" }}
{{ $d := datasource "somealias" }}
...

I think I prefer this last one since it separates concerns... This could interact nicely with the -d flag, where the commandline overrides the defineDatasource function.

What do you think?

[osterman]

Your proposal sounds wonderful. I don't have a strong preference either way on the interface. I see the benefits of having a defineDatasource, albeit for my use-case not necessary. Also, the somealias would need to be dynamic when in the context of a range - thus providing limited utility and more complicated template code. For my use-case, your first example is most preferable.

[hairyhenderson]

Thanks for the feedback, @osterman - TBH I don't see why I can't support both the bare-url approach (first example) as well as the defineDatasource approach. They're slightly different depending on the use-case.

It turns out that defineDatasource seems to be a decent way to define a default datasource URL - something that could be very useful in shared templates with shared datsources. It's annoying to have to know all the -d arguments that have to be defined in order for the template to even render!

I think I'll implement both of these.

[osterman]

It turns out that defineDatasource seems to be a decent way to define a default datasource URL

Yes, very nice indeed.

It's annoying to have to know all the -d arguments that have to be defined in order for the template to even render!

Yes, this has been a pain-point for us as well.

[osterman]

Wow! @hairyhenderson didn't expect you would implement that so quickly. Eager to go check this out. Thanks again!

[hairyhenderson]

@osterman you're welcome! 😄