Load data from YAML, JSON, CSV

[janriokrause]

It would be great to load data from YAML, JSON and CSV. Jekyll supports this feature called "Data Files". See http://jekyllrb.com/docs/datafiles/.

[Incognito]

As in a provider of variables?

[simensen]

@Incognito I believe so, yes. The way I see it, it would be a lot like generating chunks of sculpin_site.yml without putting it in sculpin_site itself.

[simensen]

I think that it would be somewhat easy to implement a DataProviderInterface implementation that reads the contents of a file. Maybe a FlatfileDataProvider that could be directed to look at sources/_data by default? And the name registered w/ the DataProviderManager could be the filename without any suffix. So sources/data/authors.(yaml|json) could be brought into a source by doing "use: - authors" or something like that?

As soon as we do this people will want global data providers in a big way; which would be nice but a lot harder to implement.

We could also inject this into site instead of using the DataProviderInterface. Anything under source/data could be injected into site.data.NAME. Then it would all be considered global data that anyone can use anywhere. I'd say this is maybe a little more confusing/inconsistent, but it would be immediately more usable and easier for people to use/grok.

[Incognito]

We may be able to use some of these tools: https://github.com/symfony/Translation/tree/master/Loader -- what we're talking about isn't massively different from translation support.

[simensen]

@dragonmantank My impression was that Fillet was a "run once" sorta thing to migrate something. My impression of this feature is that it would be an ongoing thing and be treated more or less like any other source of a Sculpin site (sculpin_site.yml + anything under source/). Have I misunderstood Fillet in some way or did I otherwise misunderstand what you were getting at here?

If not, I think that this functionality should definitely be a part of Sculpin core. I just haven't had the time to implement it or design it to the point that I could ask someone else to take a look at it.

@Incognito I think that list of tools is great but I'd be hesitant to bring the Translation package into Sculpin (as far as I know it isn't currently included). Do you think they would be reusable enough as-is for just bringing raw data in?

[dragonmantank]

@simensen My phone didn't show where it was just for storing data variables (I'm stuck with just phone today). Retracting my other comment :)

[Incognito]

@simensen I've had difficulty modifying those in the past to work with a database because they seem to insist on using files (so I had to wrap it in temp streams)... it may be possible to get symfony to break that up a bit further as I think there's a valuable generic case... however, translation may be a nice feature for sculpin if that's in the roadmap.

[devjack]

I'm looking into this at the moment, specifically to be able to use sculpin to consume and spit out nicer-than-swagger documentation (so YAML to drive both content and structure). For my case I'm thinking its better off as a plugin but I'll be exploring that (and would love thoughts on how it ties in here as well)