Blog: Transforming json data with easygen

[suntong]

Filtering

Take a look at the vast amount of data a github API may return, e.g.: https://api.github.com/search/repositories?q=go.

If there is only few key data that we want, then the first thing to do is to filter out the unwanted. Can easygen do that? Yes, of course. Given the power of Go template engine, everything is possible. However, such filtering is imperative programming if done in easygen, especially when doing vertically filtering (skipping unwanted records). So it is better to use the declarative approach, using tools like jp or jq instead. Since that is outside of the scope of this article, we'll skip it and move on to the next section.

Presenting

Take a look at the end result first, at https://github.com/go-sqlparser/current/wiki/List:-Currently-Collected-Projects:

Here is what happening behind the scene -- the mark down source code that renders above:

Note that

• the project name keyword has been repeated five times in the mark down source code.
• there is a TOC index that links directly to the detailed content. I.e., each project is repeated twice, first listed briefly in the TOC/index section, then full details in the following section.
• the PrimaryLanguage is therefore need to be repeated in two sections as well.

Such repetitive tasks are best leave to tools to automate it, and easygen is such code/text auto generating tools. The mark down code generation template is as simple as this:

$ cat repos.tmpl 
## Currently Collected Projects

### List
{{range .Data.RepositoryOwner.Repositories.Edges}}
- [{{.Node.Name}} ({{.Node.PrimaryLanguage.PrimaryLanguage}})](#{{.Node.Name}}){{end}}

### Details
{{range .Data.RepositoryOwner.Repositories.Edges}}
<a name="{{.Node.Name}}"/>**{{.Node.Name}}** ({{.Node.PrimaryLanguage.PrimaryLanguage}})  
{{.Node.Url}}  
{{.Node.Description}}
{{end}}

To put them together

So in order to come up with the above end result, I need to

• download json data via API
• filter the json data with the jp
• then present the json data in human friendly mark down format with the new easygen that can read from stdin.
• in the human friendly form, if I need to convert the long number of project size from bytes to the size in KB or MB, and I can make use of easygen's built in calculation in Go template supports.
• there are lots of other transformation support function already builtin inside easygen, check out the full list with sample usage and results here.

UPDATE:

project size added to list details:

and here is the updated code generation template:

$ cat repos.tmpl 
## Currently Collected Projects

### List
{{range .Data.RepositoryOwner.Repositories.Edges}}
- [{{.Node.Name}} ({{.Node.PrimaryLanguage.PrimaryLanguage}})](#{{.Node.Name}}){{end}}

### Details
{{range .Data.RepositoryOwner.Repositories.Edges}}{{$mb := sprintf "%.2f" (divide .Node.Size 1024)}}
<a name="{{.Node.Name}}"/>**{{.Node.Name}}** ({{.Node.PrimaryLanguage.PrimaryLanguage}}, {{.Node.Size}}KB/{{$mb}}MB)  
{{.Node.Url}}  
{{ coalesce .Node.Description "No description" }}
{{end}}